Logstash 5.0 Alpha kafka input error

Elastic Stack Logstash
1

Here is the logstash.conf.
input {
kafka {
zk_connect => "10.120.31.78:2181"
topic_id => "elk"
reset_beginning => true
}
}
output{
stdout {}
}
After run the ./logstash -f logstash.conf ,I got this error.
Unknown setting 'zk_connect' for kafka {:level=>:error}
Unknown setting 'topic_id' for kafka {:level=>:error}
Unknown setting 'reset_beginning' for kafka {:level=>:error}
fetched an invalid config {:config=>"input {\n kafka {\n zk_connect => "10.120.31.78:2181"\n topic_id => "elk"\n reset_beginning => true\n }\n}\noutput{\n stdout {}\n}\n\n", :reason=>"Something is wrong with your configuration.", :level=>:error}
How could I fix this problem?
BR

2

Logstash 5 requires Kafka 0.9.X and the config has changed: https://www.elastic.co/guide/en/logstash/5.0/plugins-inputs-kafka.html

PS in you example you are using Kafka 0.8.X settings.

3

Also, if you want Kafka 0.8 support with the 5.0 Alpha, you can uninstall the new kafka plugin and install version 2.0.6.

bin/logstash-plugin uninstall logstash-input-kafka
curl -L -O 'https://rubygems.org/downloads/logstash-input-kafka-2.0.6.gem'
bin/logstash-plugin install ./logstash-input-kafka-2.0.6.gem

https://www.elastic.co/guide/en/logstash/5.0/working-with-plugins.html#installing-local-plugins

4

This setting is good for testing, but you would want to remove it from production because it can cause log duplication. Basically any logstash instance that has that entry is going to nuke the last offset recorded by the consumer group (default group name is logstash).

5

Thanks Joe.
I use the kafka version 2.11-0.9.0.1 and changed the logstash.conf.
input {
kafka {
bootstrap_servers => "10.120.31.78:2181"
topics => "elk"

reset_beginning => true

}

}
output{
stdout {}
}

And I could run ./logstash -f logstash.conf successfully,but I can not see any data even I could see the data in kafka topic.Is there anything wrong with the logstash.conf?
BR
Felix zhang

6

It's unfortunate you removed this setting instead of writing a simple warning on it because we need this for our testing environment and since we're using kafka 0.9 we can't use the old plugin version (Using 4.1.1)