Increasing Logstash Throughput When the Codec is the Bottleneck

If Kafka is in the mix, I would split the ElastiFlow pipeline into two parts. A collector/decoder and a post-processor with Kafka in the middle. On my to-do list is a video where I explain why this is a good idea for any high volume UDP use-cases.

You will also want to use the recommendations for kafka plugin configuration which I posted here... How to slow down large amount of data coming from filebeat?

Connect with me on LinkedIn (below) if you want to discuss in more detail.

Rob

GitHub YouTube LinkedIn
How to install Elasticsearch & Kibana on Ubuntu - incl. hardware recommendations
What is the best storage technology for Elasticsearch?