Still testing but the SumoLogic http output module is working much better with these additional settings. 3 hours and heap size is 10-20 % used.
automatic_retries => 10
validate_after_inactivity => 3
request_timeout => 10
interval => 5
Thank you! I will try those options.
I would like to add some more points here.
I need to find out which pipeline is taking much CPU. So now I switched to 2 pipelines with default configuration. Each pipeline will read logs from individual log files. Now I started load test again with 40~ tps. We have observed CPU is at 45% (Without Logstash it is at 39%). Next I have started Logstash with only 1 pipeline now the CPU is at 39%, like that I did for the other pipeline too.. the CPU is still at 39%. But when I start Logstash with 2 pipelines it is going up-to 45%. I am unable to understand why it is 6% of CPU when started with 2 pipelines. Any guess or workaround ?
Is Logstash unable to keep up with one or more inputs? - Yes If I add multiple pipelines CPU is spiking
are you merely concerned that it is using CPU? - According to my tollgate systems - the allowed cpu cap is 5%, as it is using 8 to 9% sometimes - yes am concerned about cpu usage also. But this was already answered by @magnus. I am trying to more scrutinize this.
Looking at CPU utilization percentage isn't a useful metric in itself. Since Logstash's CPU usage is typically proportional to the amount of inbound events there's basically no limit on the amount of CPU it can use. What you should do is look into optimizing your configuration so it doesn't use more CPU than necessary and consider offloading the processing on a dedicated server or spread out the load on multiple servers.