Logstash high CPU usage

[mamta]

Hi,

I have installed ELK on redhat machine. ELK version is 5.6.8. I searched a lot on google and discuss elastic forum as well but not able to solve it.
I have uploaded a screenshot of CPU usage.

Logstash yaml file
https://pastebin.com/raw/maDUvgxR

Logstash logs file
https://pastebin.com/raw/DbeXkaxT

Thank you.

[magnusbaeck (Magnus Bäck)]

Well, how much is Logstash actually doing? How many events per second is it processing? What does your configuration look like?

[mamta]

Hey @magnusbaeck ,

I have already sent you the link of the logstash configuration file in the above question.
How many events per second is it processing?
I have no idea how to check it.

logstash.conf link below

https://pastebin.com/raw/i53rWAa7

Thank you.

[magnusbaeck (Magnus Bäck)]

The Logstash monitoring API should be able to report the event rate.

How big is mysqld.log? Were you processing it from scratch when you looked at the CPU usage? Was Logstash actually doing anything when the CPU usage was high?

Your grok expression is quite inefficient. You should reduce the amount of DATA and GREEDYDATA patterns you use.

[mamta]

Hi @magnusbaeck

Currently we are not running any .conf file. For investigation we kept logstash idle.
So only one thread is running.

[magnusbaeck (Magnus Bäck)]

No configuration file at all? So what about the config file you posted earlier? What configuration (if any) was in use when the CPU usage was high?

At least in the past running Logstash with no configuration files has resulted in restart loops, obviously resulting in high CPU usage since the JVM is constantly restarting.

[mamta]

Hey @magnusbaeck,

Earlier we kept that conf file which I have sent you in /etc/logstash/conf.d directory but that time also CPU usage was high so for testing purpose we removed that .conf file and kept CPU idle.

I have done a setup on one more redhat server and kept below .conf file in /etc/logstash/conf.d directory but there is also CPU usage is high.
I am using below type of .conf file.

apache-access-logstash.conf

https://pastebin.com/raw/xqBuSdD9

apache-error-logstash.conf

https://pastebin.com/raw/wyirWKSD

[magnusbaeck (Magnus Bäck)]

Have you tried using the hot threads API to get some clues about what Logstash is doing? Raising the loglevel to DEBUG and checking the Logstash log could also be useful.

[mamta]

Yes, I tried hot threads API but not able to use it. I tried by using the below command.

curl -XGET 'ipaddress:9300/_node/hot_threads?pretty'

output is
This is not a HTTP port

I also tried to check whether logstash port is running or not
9300-9600 but none of the port is running.

Can you please tell me logstash default port?
I also checked logstash logs in debug mode and saw output that is quite obvious because I haven't kept any .conf file in /etc/logstash/conf.d directory.

logtsash log
[2018-08-22T11:23:32,852][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.4"}
[2018-08-22T11:23:32,889][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/*"}
[2018-08-22T11:23:32,890][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2018-08-22T11:23:32,944][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

[magnusbaeck (Magnus Bäck)]

The default port is 9600, not 9300. That's also the port number in the last line of your log.

[mamta]

Yes, I checked it. I am not getting output of hot threads.

[magnusbaeck (Magnus Bäck)]

As I said: At least in the past running Logstash with no configuration files has resulted in restart loops, obviously resulting in high CPU usage since the JVM is constantly restarting.

So, make sure you have configuration files in place before you attempt to do anything else.

[mamta]

Hey @magnusbaeck,

As you said, I have kept at least one file in /etc/logstash/conf.d directory. Also, I have given all the permission to that user and file. Then restart the logstash service but I am getting some error like in the attached screenshot.

Please help it out.

[magnusbaeck (Magnus Bäck)]

This is really weird. I don't know what's going on.