Default codecs are different for 3.0 Kafka plugins

[sundar (Sundar Rajan)]

Hello Logstash Team!
Thanks for your help in pointing me to the correct plugin versions which support Kafka 0.9 in this topic:

Logstash 2.3.1 kafka plugins ssl support

Hello, We use Kafka input and output plugins with Logstash in our log pipeline. We are using 2.3.1 version of Logstash and Elasticsearch. I see that there is now Kafka 0.9 producer/consumer support in LS 5.0. Any chance you would backport that to 2.3.1? Or is there a way I can retrofit to use that plugin with my Logstash 2.3.1? We are using Kafka 0.9, and are guaranteed not to have any 0.8 brokers, so backward compatibility is not an issue for me. We have secured with the pipeline from filebe…

One peculiar thing we noticed (after much hair-pulling on why field extraction with KV filters were broken after we moved to 0.9 version of the plugin) is that the default codec for Kafka output plugin and Kafka input plugin, are different.

Can you please fix this in releases going forward? The defaults (IMO) should be JSON for both. As such we fixed it easily by forcing JSON codec in both output and input sides.

Thanks
-Sundar

[allenmchan (Allen Chan)]

In my testing, the kafka input throughput was a lot lower with json codec enabled.
You might want to do some tests and see if it is the same in your workload.

[sundar (Sundar Rajan)]

Thanks @allenmchan. Are there any benchmarks that you can share around this performance test? Would love to try and repro the same on our side.

Rgds

[allenmchan (Allen Chan)]

All i have are these notes from my test in february

"So for the benchmark i am using kafka-producer-perf.sh

• using record-size 20000 bytes
  producer was able to push 1000 messages per second and the logstash consumer was able to match that (only with codec=> plain. With default json codec, performance dropped to 200 messages per second)

• using record-size 1000000 bytes (1MB)
  producer was only able to push ~250-300 messages per second. And logstash consumer was very low in consumption too. Seems to be bounded by network. Increasing all the settings didnt increase the records/sec nor network mb/sec"