Having an issue connecting to Kafka 0.9 from 5.0 alpha4 release
Getting following error:
"Exception in thread "Ruby-0-Thread-13: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.0/lib/logstash/inputs/kafka.rb:162" org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'topic_metadata': Error reading array of size 148300, only 30 bytes available"
When searching for that error, all problems seem to relate to version mismatch of client and kafka versions.
The 5.0 alpha 4 docs say that this release works with Kafka 0.9
I'm wondering whether its just the docs that need updating and it should say Kafka 0.10 that it works with ?
I regressed logstash to 2.3 and it connected to Kafka no problems 
Does anyone know ? Does logstash 5.0 only work with Kafka 0.10 ?
Many Thanks,
Mick
My understanding with the kafka clients is that forward versions are not compatible. Meaning you cannot connect a 0.10 client to a 0.9 broker.
The other way around works (backward versions). 0.10 version of broker supports 0.9 clients.
So is logstash 5.0 a kafka 0.10 client ?
Thanks,
Mick
I understand it now - not sure if the docs have just been updated since I logged this, or whether I missed it before.
different plugins are compatible with different kafka versions.
Logstash/kafka plugin v4.x.x talks to Kafka 0.9
Logstash/kafka plugin v5.x.x talks to Kafka 0.10
Logstash 5.0 alpha 4 ships with plugin Logstash/kafka plugin v5.0.0 and as such that is why I had the issues. The fix would be to uninstall logstash/kafka plugin v5.0.0 and install logstash/kafka plugin v4.x.x
Cheers,
Mick
Yes they added a compatibility matrix that hopefully clarifies it for everyone.