Logstash high cpu usage when idle

[alexus]

I'm running ELK stack:

elk & beats:

# rpm -q elasticsearch logstash kibana
elasticsearch-6.0.1-1.noarch
logstash-6.0.1-1.noarch
kibana-6.0.1-1.x86_64
# rpm -q filebeat metricbeat
filebeat-6.0.1-1.x86_64
metricbeat-6.0.1-1.x86_64
# 

My environment:

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.4 (Maipo)
# uname -a
Linux X 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 13 10:46:25 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
# 

and I noticed that my CPU is being utilized:

# top -n 1 | head

top - 11:01:00 up 3 days, 23:21,  2 users,  load average: 4.30, 4.19, 4.21
Tasks: 157 total,   1 running, 156 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  3.2 sy, 95.2 ni,  1.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  8008592 total,   249368 free,  1972212 used,  5787012 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  5415744 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                              
 2559 logstash  39  19 2908216 224376  13032 S 362.5  2.8   0:27.85 java                                 
    1 root      20   0  193712   6872   4092 S   0.0  0.1   2:09.25 systemd                              
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.08 kthreadd                             
# 

elasticsearch log:

# tail -1 /var/log/elasticsearch/x.log
[2017-12-13T10:52:13,947][INFO ][o.e.m.j.JvmGcMonitorService] [X] [gc][341727] overhead, spent [340ms] collecting in the last [1s]
# 

My ELK stack is not being super utilized.

I did noticed that is CPU that being used is in nice CPU, so maybe it's not as bad?

Please advise.

[alexus]

in my case it was related to logstash rather then elasticsearch...

after I took a look at /var/log/logstash/logstash-plain.log, I saw logstash had an issue writing to file/directory and that somehow generated this huge overhead with cpu (at least it was nice)

after I addressed writing issue and restart logstash, cpu went down and now it's idle:

# top -n 1 | head

top - 11:30:42 up 3 days, 23:50,  1 user,  load average: 0.06, 0.23, 1.46
Tasks: 152 total,   1 running, 151 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.2 us,  1.6 sy,  0.0 ni, 92.1 id,  3.2 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  8008592 total,   433696 free,  2055752 used,  5519144 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  5332192 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                              
 1095 root      20   0 1316972  62472  19240 S  13.3  0.8  27:30.18 dockerd-latest                       
    1 root      20   0  193712   6872   4092 S   0.0  0.1   2:09.75 systemd                              
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.08 kthreadd                             
#