Logstash (logstash 2.2.4) failing to start taking Kafka Input with bootstrap_servers option


(Rajendra Naidu) #1

With reference to "Modern Kafka clients do not connect to Zookeeper. Historical versions of Kafka exposed Zookeeper to clients to retrieve the full broker list (Kafka 0.7) and the initial consumer groups were implemented in the clients using Zookeeper (Kafka 0.8). But since Kafka 0.8 (and 0.9 for consumer groups) there is no longer any need for clients to connect to Zookeeper, the metadata and consumer group functionality has moved to the Kafka broker. In fact, it is highly recommended not to make Zookeeper reachable from clients due to security aspects."

Tried to configure Logstash to take input from Kafka Cluster using bootstrap_servers option as follows:

input {
kafka {
topic_id => "ericsson"
bootstrap_servers => "192.168.4.79:9092"
type => "kafka-input"
}
}

which resulted in the following error
[root@elk ~]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf
Unknown setting 'bootstrap_servers' for kafka {:level=>:error}
Error: Something is wrong with your configuration. {:level=>:error}
[root@elk ~]#

When the same thing is run with Zoo Keeper option as follows it does work well

input {
kafka {
topic_id => "ericsson"
zk_connect => "192.168.4.79:2181"
type => "kafka-input"
}
}

Please provide your inputs

Thanks and Regards,
Rajendra


(Magnus B├Ąck) #2

As documented the kafka input that works with Logstash 2.2.4 doesn't support the bootstrap_servers option. You need to upgrade your Logstash.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.