Logstash 6.2.2 high CPU usage

Elastic Stack Logstash
1

I upgraded Elastic Stack from 5.6.3 to 6.2.2, but the Logstash CPU usage go high and no error logs.

Start command:

./logstash -f '/data/conf/main.conf' -l /data/logs/logstash/

Logstash config:

input {
  beats {
    port => 5044
    add_field => {
      "logstashId" => "logstash1"
    }
  }
}

output {
  elasticsearch {
    action => "index"
    hosts  => ["localhost:9200"]
    index  => "test-%{+YYYY.MM.dd}"
    manage_template => false
    template_overwrite => true
    document_type => "%{[@metadata][type]}"
  }
}

Logstash stats:

image
image.png1350×686 77.3 KB

$top

top - 09:40:02 up 371 days, 3 min,  2 users,  load average: 1.74, 1.49, 1.28
Tasks: 190 total,   1 running, 189 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.5 us,  0.1 sy,  0.0 ni, 99.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 20396440 total,  8862336 free,  6573776 used,  4960328 buff/cache
KiB Swap: 16515068 total, 15727996 free,   787072 used. 13406740 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
26113 admin  20   0 11.969g 1.453g  20304 S 100.0  7.5   1119:41 java
    1 root      20   0   74020  35220   1332 S   0.0  0.2   6:19.52 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:04.59 kthreadd

$ top -H -p 26113

top - 09:40:18 up 371 days, 3 min,  2 users,  load average: 1.53, 1.46, 1.27
Threads:  64 total,   1 running,  63 sleeping,   0 stopped,   0 zombie
%Cpu(s): 26.0 us,  1.8 sy,  0.0 ni, 72.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 20396440 total,  8861800 free,  6574300 used,  4960340 buff/cache
KiB Swap: 16515068 total, 15727996 free,   787072 used. 13406216 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
26154 admin  20   0 11.969g 1.453g  20304 R 96.3  7.5 993:23.30 LogStash::Runne
26126 admin  20   0 11.969g 1.453g  20304 S  2.7  7.5  24:44.55 java
26127 admin  20   0 11.969g 1.453g  20304 S  2.7  7.5  24:44.69 java
26128 admin  20   0 11.969g 1.453g  20304 S  2.7  7.5  24:44.26 java
26129 admin  20   0 11.969g 1.453g  20304 S  2.7  7.5  24:44.46 java

What could be possible reason for high CPU usage?

2

Do you have 'config.reload.automatic' set to 'true' in your logstash.yml?

Disabling the automatic config reload resolved the issue for us:

config.reload.automatic: false

3

Yes, I set 'config.reload.automatic' to 'true' in logstash.yml, but the root cause is not here.
The value of "config.reload.interval" is not same between 5.6 and 6.2:

5.6:

config.reload.interval : 3

6.2

config.reload.interval : 3s

so it reload the config files all the time. after the value changed from '3' to '3s', the CPU usage go down.

3 Likes
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.