Hi community!
We are using ELK-Stack for a couple of years having a Logstash Shipper / Filter architecture (currently 2 Shipper nodes with 6 Filter nodes). With the latest updates from version 6.2 to 6.8 we've discovered a light performance increase on filters and a massive performance decrease on Shippers. Consider our test data:
- Version 6.2.4 (java8) can handle up to 8000 messages / s (we use a syslog generator to process data)
- Version 6.8.8 (java11, java8 is even worse) can handle up to 4000 messages / s
We were quite surprise by this impact (having tested filter performance in advance not much on shipping). So our question is, does anyone have an idea how to tweak for processing more on shipper level (no filters in place)?
What we've tested so far:
- change UDP queue_size (even queue_size of 10000 packages will cause data loss when sending only 5000 msgs)
- changed settings of pipeline.batch_size and workers (not expected much change since there are no filters)
by checkig changelogs we could not identify any reason on udp_input plugin side and we have confirmed this by using versions with old/new input_plugin (same behaviour). We suspect a change in behaviour on internals which is not obvious for us.
We can confirm the issue by using netstat -s --udp (errors couning up with 6.8.8 and sending 8000 messages /s but not wich version 6.2.4)
Any hint would be greatly appreciated!
Thanks,
Rolf