Should I expect 2.3 kafka plugins to work with kafka 0.10?

Elastic Stack Logstash
1

All our other kafka 0.9 clients work fine with 0.10 brokers. But whenever I start a kafka-output logstash hangs, and can only be recovered with kill -9. Nothing in the logs. (Have not run with -debug yet.)

kafka-input does not seem to have the same effect. But I don't really know if it's working at all as no input can send logs.

2

Just noticed that I have v 2.x of the plugins install. Seems I should have 3.x. Trying that now...

Nope, same behavior with 3.x plugins.

3

But I do know the plugin is starting, from logstash.err:

log4j:WARN No appenders could be found for logger (org.apache.kafka.clients.producer.ProducerConfig).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

4

The logstash team put together a nice compatibility matrix on the logstash plugin page

Do you see any connections from logstash to kafka when running something like netstat?

5

I had two problems, both user error.

The first was the EC2 security groups did not allow egress to kafka. The second, I was using the docs released docs, not the master docs. It seems that 3.x has unified the plugin parameters between input and output.

Just a few minutes ago we finished updating the production environment from rabbit to kafka and all is good.

thanks!