Upgrade Logstash 5 -Kalfka plugin issue

Elastic Stack Logstash
1

I am upgrading logstash from 2.3 to 5.1 . I am facing issue with kalfka input plugin. I have tried to remove the 5.1 plugin and install lower version but it is failing due to other dependencies.Also i have tried to change the input pattern to compatible to new plugin version. But looks like new plugin doesnt connect with zookeeper. can anyone suggest alternatives . please find my input pattern below.

input {
kafka {
zk_connect => 'x:x:x:x:2181,x:x:x:x:2181,x:x:x:x:2181'
white_list => 'SubmitOrder,Conversion,Session,Navigation'
consumer_id => 'hostname'
consumer_threads => 4
}
}

2

If you are using Logstash 5.x, then use the below:

input {
  kafka {
    bootstrap_servers => "xx.xx.xx.xx,yy.yy.yy.yy,zz.zz.zz.zz"
    topics => ["abc","def"]
    #decorate_events => true
  }
}
3

Hi ,

thanks i have tried that too. looks like we cannot use zookeeper with logstash 5x and even we need to upgraede kafka to compatible with logstash 5x

4

Didn't it work? Which version of Kafka are you using. I used kafka_2.10-0.10.1.0. So yes, you'd need to upgrade your kafka. The logstash input plugins doc should tell you which version would work with which.