Hello. I just started using Logstash 5, but I am stuck, and cannot figure out why.
So if anybody have any idea/suggestions please let me know.
I use Logstash to capture netflow V9 from UDP ports and forward it towards my Elasticsearch cluster.
In my setup I have two identical Logstash nodes running 2.4.0, that each collect around 1.6mil flows/min.
I installed Logstash 5 on one of them, and configured both the config file, jvm.options and logstash.yml, but for some reason hit a 150k flows/min roof.
I can see from htop that Logstash are able to allocate 50g memory, but the CPU cores are mostly idle.
From having the same setting as the other Logstash node, I have been scaling individually on everything I could find, with no luck.
From another post i saw something about number of open files, but since my setup reads from an UDP port, I figured that it might not be it.
I myself think the problem was with the number of workers or number of workers/batch size(and delay), but changes on them have shown no effect.
So I must be missing something.
Ideas?
Best regards Simon.
many thanks. I removed the workers, queue_size and flush_size from the config file at installation due the changing to 5.0, but i should have had let the workers stayed.