Hi,
I am sending data from kafka cluster to logstash(indexer). I have 3 kafka servers of which one acts as the leader and logstash sends the data to the leader.
Now I changed the leader (by shuting down the kafka leader which is now the slave). Kafka automatically changes it's leader to other server in the cluster but the logstash is now not able to recieve the data.
The config of logstash
input {
kafka {
zk_connect => "x.x.x.1:1,x.x.x.2:2, x.x.x.3:3"
topic_id => "logstash"
codec => "json"
}
}
output {
stdout {}
elasticsearch {
hosts => "x.x.x.x:4
index=> "myindex"
}
}
Now if I change back the leader (to the previous one) logstash again start recieving the data.
I am not able to find out where exactly the problem is? Can't Logstash recognize the new leader in the kafka cluster? Or is there any configuration which needs to be done at logstash?