Does Logstash 2.1.1 work well with Kafka 0.9.0?

Kenny_Qiao · January 11, 2016, 5:15am

I plan to upgrade Kafka to 0.9.0 and wondering if logstash 2.1.1 could work well with Kafka 0.9.0.
Does anyone have experience using logstash 2.1.1 with Kafka0.9.0 in production?

Thanks.

warkolm · January 11, 2016, 1:22pm

It's not ready as of this moment - https://github.com/logstash-plugins/logstash-input-kafka/issues/48

mikesparr · January 11, 2016, 7:12pm

Works fine here and pushing soon to production.

Beats -> Logstash -> Kafka <- Logstash -> ES

I'm using it in staging and consumed 150 million events from Beats (top, file, packet) in past few days without issue, published to Kafka topics, and another Logstash instance consumed topics and published to ES. Upgraded ES and LS to 2.1.1, added Beats, and Confluent Platform 2.0 (with Kafka 0.9.0). No issues.

Joe_Lawson · January 11, 2016, 9:41pm

Thanks for the feedback!

allenmchan · January 13, 2016, 1:39am

@mikesparr, Can you share in general your hardware specs?