Default codecs are different for 3.0 Kafka plugins

Hello Logstash Team!
Thanks for your help in pointing me to the correct plugin versions which support Kafka 0.9 in this topic:

One peculiar thing we noticed (after much hair-pulling on why field extraction with KV filters were broken after we moved to 0.9 version of the plugin) is that the default codec for Kafka output plugin and Kafka input plugin, are different. :frowning:

Can you please fix this in releases going forward? The defaults (IMO) should be JSON for both. As such we fixed it easily by forcing JSON codec in both output and input sides.


In my testing, the kafka input throughput was a lot lower with json codec enabled.
You might want to do some tests and see if it is the same in your workload.

Thanks @allenmchan. Are there any benchmarks that you can share around this performance test? Would love to try and repro the same on our side.


All i have are these notes from my test in february

"So for the benchmark i am using

  • using record-size 20000 bytes
    producer was able to push 1000 messages per second and the logstash consumer was able to match that (only with codec=> plain. With default json codec, performance dropped to 200 messages per second)

  • using record-size 1000000 bytes (1MB)
    producer was only able to push ~250-300 messages per second. And logstash consumer was very low in consumption too. Seems to be bounded by network. Increasing all the settings didnt increase the records/sec nor network mb/sec"

1 Like