Logstash Consuming high CPU

Elastic Stack Logstash
1

Team,
Below is my CPU configuration

CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             8
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 79

I am trying to start multiple pipelines with -r option.

Below is my pipelines configuration.

- pipeline.id: LOG1
  queue.type: persisted
  path.config: "./config/log1.conf"
- pipeline.id: LOG2
  queue.type: persisted
  path.config: "./config/log2-logstash.conf"
- pipeline.id: LOG3
  queue.type: persisted
  path.config: "./config/log3-logstash.conf"
- pipeline.id: LOG4
  queue.type: persisted
  path.config: "./config/log4-logstash.conf"
- pipeline.id: LOG5
  queue.type: persisted
  path.config: "./config/logstash_elg.conf"

It is observed that CPU usage is spiking sometimes to 40 to 50%.

Thanks

2

Have you profiled your Java Heap ? My CPU went down after I applied the right heap size.

Xms represents the initial size of total heap space

Xmx represents the maximum size of total heap space

#-Xms256m
-Xms3g
-Xmx3g

3

Logstash will attempt to process incoming events as fast as possible, so if you receive a bunch of messages at once it's normal and expected for Logstash to use 50% CPU.

1 Like
4

Still testing but the SumoLogic http output module is working much better with these additional settings. 3 hours and heap size is 10-20 % used.
automatic_retries => 10
validate_after_inactivity => 3
request_timeout => 10
interval => 5

5

Thank you! I will try those options.
I would like to add some more points here.
I need to find out which pipeline is taking much CPU. So now I switched to 2 pipelines with default configuration. Each pipeline will read logs from individual log files. Now I started load test again with 40~ tps. We have observed CPU is at 45% (Without Logstash it is at 39%). Next I have started Logstash with only 1 pipeline now the CPU is at 39%, like that I did for the other pipeline too.. the CPU is still at 39%. But when I start Logstash with 2 pipelines it is going up-to 45%. I am unable to understand why it is 6% of CPU when started with 2 pipelines. Any guess or workaround ?

Please let me know if you need any other details

Thanks.

6

@magnusbaeck @PandKing

Till now we are using Elasticsearch output, now we are moving to syslog/tcp. Can I expect a better performance now?

7

What is your performance problem, specifically? Is Logstash unable to keep up with one or more inputs? Or are you merely concerned that it is using CPU?

40tps seems exceptionally slow (a bare input/output pipeline with minimal filtering is typically in the thousands to tens-of-thousands events per second).

  • What do your pipelines look like?
  • Which plugins do you use?
  • Do you really heavily on grok patterns, and if so, are you using many patterns, and are your patterns properly anchored?
  • do you enrich from external sources, such as with the DNS filter?
  • is your Elasticsearch output tuned to appropriate batch sizing for your use?

All of these things can have a huge affect on performance and thoroughput.

1 Like
8

@yaauie

Ok.
As I have stated before w/o logstash our cpu consumption is at ~39%, when I add logstash with single pipeline it is at ~39%, but when I add multiple pipelines like 2 or more it is going to ~45%.

What do your pipelines look like? - Default configuration

Which plugins do you use? - If the question is about ruby codes, yes we use Nokogiri XML plugin if there is any xml in log line, if json is there we use ruby json code

Do you really heavily on grok patterns, and if so, are you using many patterns, and are your patterns properly anchored? Yes we use grok patterns heavily. And we are also trying to anchor the patterns

do you enrich from external sources, such as with the DNS filter? - No external sources. Read from Log file, enrich or mask log line, put it in Elasticsearch

is your Elasticsearch output tuned to appropriate batch sizing for your use? - Our Elasticsearch instance is maintained by an enterprise team. Batch sizing am not sure of it.

Please let me know if you need any specific details. I am ready to share my config file but am afraid thats a compliance issue.

Thanks.

9

I'll repeat two questions from @yaauie that you haven't answered:

What is your performance problem, specifically? Is Logstash unable to keep up with one or more inputs? Or are you merely concerned that it is using CPU?

10

Is Logstash unable to keep up with one or more inputs? - Yes If I add multiple pipelines CPU is spiking

are you merely concerned that it is using CPU? - According to my tollgate systems - the allowed cpu cap is 5%, as it is using 8 to 9% sometimes - yes am concerned about cpu usage also. But this was already answered by @magnus. I am trying to more scrutinize this.

11

Looking at CPU utilization percentage isn't a useful metric in itself. Since Logstash's CPU usage is typically proportional to the amount of inbound events there's basically no limit on the amount of CPU it can use. What you should do is look into optimizing your configuration so it doesn't use more CPU than necessary and consider offloading the processing on a dedicated server or spread out the load on multiple servers.

2 Likes
12

@magnusbaeck @yaauie

We see a better performance in PROD servers compared to lower environments. Thanks for all your suggestions.

Thank you both!!!

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.