Logstash high CPU

[jpeppard (Jacob Warr)]

Hi all. Just installed ELKstack v8.4.1 on ubuntu server 20 to play around with.

All is well with the data itself, Metricbeat is sending data to Elasticsearch from two other Ubuntu server machines. Visualizations and data ingestion is working fine.

My issues with Logstash. Currently logstash is ingesting no data. The Metricbeat apps are configured to send to Elasticsearch directly.

For some reason I am seeing a constant CPU pegging from Logstash with low priority CPU threads.
This happens whether the jvm.options heap size is 1, 3, or 4 gigabytes.

Is this normal behavior when no data is even being ingested by logstash?

[leandrojmp (Leandro Pereira)]

Do you have anything in Logstash logs?

Also, what is your configuration? You need at least one working configuration or logstash would not even be running.

Another issue is that your machine is too small to run both Logstash and Elasticsearch, per default Elasticsearch will use half of your RAM, so you cannot set the JVM Heap for Logstash to 4 GB as you did.

Since you are just testing, I would recommend that you set the RAM for Elasticsearch JVM Heap to 2 GB and at 512 MB or 1 GB maximum to Logstash.

[jpeppard (Jacob Warr)]

Thanks for your help Leandro!

I have logstash setup as an automatic start service via systemctl. I have only one conf test file with the below config. It does not correspond to any input.

input { stdin { } }

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }

I have updated the java heap, allocated 2GB to Elastic Search and 1GB to Logstash as you recommended.

Here is the output of logstash log after applying the new JVM values and restarting. Same behavior according to htop

[2022-09-12T23:18:52,402][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2022-09-12T23:18:52,411][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.4.1", "jruby.version"=>"jruby 9.3.6.0 (2.6.8) 2022-06-27 7a2cbcd376 OpenJDK 64-Bit Server VM 17.0.4+8 on 17.0.4+8 +indy +jit [x86_64-linux]"}
[2022-09-12T23:18:52,413][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-09-12T23:18:54,409][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-09-12T23:18:54,902][INFO ][org.reflections.Reflections] Reflections took 100 ms to scan 1 urls, producing 125 keys and 434 values
[2022-09-12T23:18:55,530][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2022-09-12T23:18:55,585][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2022-09-12T23:18:55,851][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2022-09-12T23:18:56,014][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2022-09-12T23:18:56,027][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.4.1) {:es_version=>8}
[2022-09-12T23:18:56,030][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2022-09-12T23:18:56,083][INFO ][logstash.outputs.elasticsearch][main] Config is compliant with data streams. `data_stream => auto` resolved to `true`
[2022-09-12T23:18:56,087][INFO ][logstash.outputs.elasticsearch][main] Config is compliant with data streams. `data_stream => auto` resolved to `true`
[2022-09-12T23:18:56,093][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2022-09-12T23:18:56,121][WARN ][logstash.filters.grok    ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2022-09-12T23:18:56,148][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2022-09-12T23:18:56,374][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/test.conf"], :thread=>"#<Thread:0x3775f1e9 run>"}
[2022-09-12T23:18:57,086][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.71}
[2022-09-12T23:18:57,160][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-09-12T23:18:57,262][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2022-09-12T23:18:58,085][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2022-09-12T23:18:58,349][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
[2022-09-12T23:18:58,432][INFO ][logstash.runner          ] Logstash shut down.

[Badger]

@jpeppard

input { stdin { } }

If you are running logstash as a service then stdin will be piped to /dev/null. When logstash reads EOF from that it shuts down. The service manager may then immediately restart it and it will use perhaps a minute of CPU to get started. Then it will read EOF and shut down, only to be restarted again. Does the process id change every couple of minutes?

[jpeppard (Jacob Warr)]

The PID does change frequently and the logs indicate that the service is restarting periodically. I removed the stdin line from my config but still the same behavior.

I will have to start sending some input to logstash to see if it stabilizes. Otherwise I suppose I can just disable the service for now.

[leandrojmp (Leandro Pereira)]

@jpeppard

I removed the stdin line from my config but still the same behavior.

What did you put in the place? Logstash needs a working config, if you do not have an input it will probably enter in a restart loop again and your CPU will be high.

If you are not using Logstash just disable the service.

If you want to keep it running just add a dummy file input.

Something like:

input {
    file {
        path => "/tmp/path-to-a-non-existent-file"
    }
}

[jpeppard (Jacob Warr)]

Thanks, I have added this and the cpu usage has calmed down completely.

Thanks both for your help!