Logstash kafka input issues in reconnecting to zookeeper

Elastic Stack Logstash
1

We are using logstash version : logstash-2.3.3

Right now we are using kafka input to read messages from broker which is version 0.9


{
 
  kafka {
    topic_id => "{{kafka_log_consumer_topic_id}}"
    group_id => "{{kafka_consumer_group_id}}"
    zk_connect => "{{zk_consumer_connect_url}}"
  }
 
}
 
filter{
if  [level] == "ERROR" or [level] == "WARN" or [level] == "FATAL" {
 
mutate{
  add_field => {
    "thrownMessage" => "%{[thrown][message]}"
    "applicationName" => "%{[contextMap][applicationName]}"
    "errorId" => "%{[contextMap][errorId]}"
   }
}
uuid{
   target => "logEventId"
}
}
}
 
output {
 
if  [level] == "ERROR" or [level] == "WARN" or [level] == "FATAL" {
 
  elasticsearch {
    hosts => "{{elastic_search_hosts}}"
  }
 
  kafka {
    topic_id => "{{kafka_error_publish_topic}}"
    bootstrap_servers => "{{kafka_error_publish_host}}"
  }
 
}
else {
  elasticsearch {
    hosts => "{{elastic_search_hosts}}"
  }
}
}

Sometimes when our zookeeper goes down , and gets back up after sometime logstash is not reconnecting back again.

Does logstash kafka input supports using the new kafka consumer API 's which is not dependent on zookeeper ?

2

wanted to know in case if we can refresh zookeeper connections in case of zookeeper failure

3

The input does support the new APIs but you need to use the right plugin version. Check https://github.com/logstash-plugins/logstash-input-kafka#kafka-compatibility for reference.

4

The matrix says I need to use 2.4 of logstash and I and don't find that anywhere , even in git I see only 2.3.X and version 5.0.X but not the 2.4.

0.9 2.4, 5.0 4.0.0 4.0.0

5

I see there is a 2.4 branch but not sure where to download the same

6

Yeah it is confusing. The plugins are really rubygem packs. So the matrix says for Logstash 2.3.3 you should use 3.0.0 and it looks like the latest logstash-input-kafka 3.0.x is 3.0.3 https://rubygems.org/gems/logstash-input-kafka/versions/3.0.3.

so you need to use the logstash-plugin tool and install that version.

logstash-plugin install --version 3.0.3 logstash-input-kafka

Those have the bootstrap_servers you want: https://github.com/logstash-plugins/logstash-input-kafka/blob/v3.0.2/lib/logstash/inputs/kafka.rb#L51