Dear everyone!
I have a logstash get log from 3 instances apache kafka and zookeeper cluster.
When I test turn off the Leader zookeeper from cluster. Logstash still connect to OLD broker ID of OLD Leader, And traffic cannot show to Kibana.
Please tell me how to change the new brokder ID, when Leader zookeeper fail.
Here is my apache.conf of logstash
input {
kafka {
bootstrap_servers => '10.0.1.48:9092,10.0.1.49:9092,10.0.1.50:9092'
topics => ["dns3000"]
consumer_threads => 3
auto_offset_reset => "earliest"
decorate_events => "true"
request_timeout_ms => "300"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
geoip {
source => "clientip"
}
}
output {
elasticsearch {
hosts => ["10.0.1.43:9200"]
}
}
Here is logs of logstash
Aug 24 10:36:46 elk-logstash logstash[2834]: [2019-08-24T10:36:46,815][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-0, groupId=logstash] Connection to node -3 (/10.0.1.50:9092) could not be established. Broker may not be available.
Thanks!