Suspected memory leak leading to heap OOM in Logstash 5.6.6


(Christopher Hodges) #1

We upgraded to Logstash 5.6.6 across our estate and soon saw Logstash memory usage climb until failure with OOM heap errors on all boxes. The heavier the load the quicker the failure. Each server runs a specific config for its role.

Downgrading to 5.6.5 fixed the problem.

You can see the issue reflected in Memory and CPU Usage on these Nginx proxies. Logstash was upgraded on the 22/01, restarted a couple of times and then downgraded on the 30/01, where the line clearly flattens out to where it was prior to the upgrade.


On these proxies, 2 were running v5.6.6 and 2 v5.6.5. It's clear which 2.


We took a look at the heap dump, here's a screenshot of the overview.

We saw Logstash recover a few times, and sometimes carry on processing at full heap usage, but at other times it stopped processing without recovery or the process stopping.
This was on Ubuntu 16.04.3 LTS.


(Jake Landis) #2

Conversation is continuing over at https://github.com/elastic/logstash/issues/9127


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.