Logstash CPU usage issue after some time running

After some time of operation, our logstash instances (same configuration) are experiencing CPU and load average issues as shown below.


When the issue occurs we have to restart the impacted logstash instance for it to behave normally again (on the above graph, it has been restarted at the vertical bar) for about 24H until the next issue occurs.
If we delay the restart between two instances, the issue is delayed by about the same time so it does not look like to be related to the ingested events.

We are running logstash 5.6.8 from the rpm repository.
With the following additional plugins:

  • x-pack
  • logstash-filter-translate

And the following configuration:

  • jvm.options


  • logstash.yml

node.name: logstash1
path.data: /var/lib/logstash
pipeline.workers: 8
pipeline.batch.size: 250
pipeline.batch.delay: 5
path.config: /etc/logstash/conf.d
log.level: info
path.logs: /var/log/logstash
xpack.monitoring.enabled: true
xpack.monitoring.collection.interval: 10s
xpack.monitoring.elasticsearch.url: ["http://elastic1:9200","http://elastic2:9200"]
xpack.monitoring.elasticsearch.username: xpack_monitoring
xpack.monitoring.elasticsearch.password: *********

And no error messages in logstash-plain.log.

We were doing lots of updates in the filters and had to restart the instance quite often so we did not notice the issue until after some time.
In the meantime we have also updated to 5.6.8 and installed xpack monitoring.
I don't know exactly which of the previous action caused the issue.
It worked without issue for about a year before that.

Any idea ?

I tried updating my logstash instances to version 6.2.3 from version 5.6.8 with the exact same configuration and plugins.
Everything is now working fine: no more high CPU/load average, etc...

And no side effects in forwarding logs from logstash 6.2.3 to elasticsearch 5.6.8.

There is definitely an issue with logstash version 5.6.8.

Below the graphs now that the logstash instance is working as intended

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.