Hello Logstash Team!
Thanks for your help in pointing me to the correct plugin versions which support Kafka 0.9 in this topic:
One peculiar thing we noticed (after much hair-pulling on why field extraction with KV filters were broken after we moved to 0.9 version of the plugin) is that the default codec for Kafka output plugin and Kafka input plugin, are different.
Can you please fix this in releases going forward? The defaults (IMO) should be JSON for both. As such we fixed it easily by forcing JSON codec in both output and input sides.
using record-size 20000 bytes
producer was able to push 1000 messages per second and the logstash consumer was able to match that (only with codec=> plain. With default json codec, performance dropped to 200 messages per second)
using record-size 1000000 bytes (1MB)
producer was only able to push ~250-300 messages per second. And logstash consumer was very low in consumption too. Seems to be bounded by network. Increasing all the settings didnt increase the records/sec nor network mb/sec"