5.6.8 java.lang.OutOfMemoryError after 6-10 days

[stefws (Steffen Winther Sørensen)]

Since we patched from 5.6.5 to 5.6.8 we're seeing logstash memory leakage so it'll now run out of memory after 6-10 days, nothing changed in our filters etc. So assuming there's a leak somewhere maybe. Before memory usage was steady (see memory trend graph).

Anyone else seeing this or know of a possible issue/workaround/fix, please let me know.

TIA

# rpm -qi logstash
Name        : logstash                     Relocations: / 
Version     : 5.6.8                             Vendor: Elasticsearch
Release     : 1                             Build Date: Fri 16 Feb 2018 07:11:44 PM CET
Install Date: Thu 22 Feb 2018 12:26:39 PM CET      Build Host: packer-virtualbox-iso-1518356860

...

[yaauie (Ry Biesemeyer)]

Any info you're willing to provide about how your pipelines are configured, including which plugins they use (and what versions), heap settings, how many workers, whether you've got persistent queueing enabled, etc, will be helpful in narrowing down the surface area of the problem.

[stefws (Steffen Winther Sørensen)]

Whatever you desire... pipelines are processing req.s from filebeat/metricbeats also now 5.6.8 and collectd.

JVM options:

# awk '!/^#/ && NF!=0{print}' jvm.options 
-Xms512m
-Xmx4g
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+DisableExplicitGC
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-XX:+HeapDumpOnOutOfMemoryError

Logstash config:

# awk '!/^#/ && NF!=0{print}' logstash.yml 
path.data: /var/lib/logstash
pipeline.workers: 8
pipeline.output.workers: 2
pipeline.batch.size: 125
pipeline.batch.delay: 5
path.config: /etc/logstash/conf.d
log.level: info
path.logs: /var/log/logstash

Installed plugins (not all used):

# cd /usr/share/logstash; bin/logstash-plugin list --verbose
logstash-codec-cef (4.1.4)
logstash-codec-collectd (3.0.8)
logstash-codec-dots (3.0.6)
logstash-codec-edn (3.0.6)
logstash-codec-edn_lines (3.0.6)
logstash-codec-es_bulk (3.0.6)
logstash-codec-fluent (3.1.5)
logstash-codec-graphite (3.0.5)
logstash-codec-json (3.0.5)
logstash-codec-json_lines (3.0.5)
logstash-codec-line (3.0.8)
logstash-codec-msgpack (3.0.7)
logstash-codec-multiline (3.0.9)
logstash-codec-netflow (3.11.0)
logstash-codec-plain (3.0.6)
logstash-codec-rubydebug (3.0.5)
logstash-filter-alter (3.0.2)
logstash-filter-cidr (3.1.2)
logstash-filter-clone (3.0.5)
logstash-filter-csv (3.0.8)
logstash-filter-date (3.1.9)
logstash-filter-dissect (1.1.4)
logstash-filter-dns (3.0.7)
logstash-filter-drop (3.0.5)
logstash-filter-fingerprint (3.1.2)
logstash-filter-geoip (4.3.1)
logstash-filter-grok (4.0.2)
logstash-filter-json (3.0.5)
logstash-filter-kv (4.0.3)
logstash-filter-metrics (4.0.5)
logstash-filter-mutate (3.2.0)
logstash-filter-ruby (3.1.3)
logstash-filter-sleep (3.0.6)
logstash-filter-split (3.1.6)
logstash-filter-syslog_pri (3.0.5)
logstash-filter-throttle (4.0.4)
logstash-filter-translate (3.0.4)
logstash-filter-urldecode (3.0.6)
logstash-filter-useragent (3.2.2)
logstash-filter-uuid (3.0.5)
logstash-filter-xml (4.0.5)
logstash-input-beats (3.1.26)
logstash-input-couchdb_changes (3.1.4)
logstash-input-dead_letter_queue (1.1.2)
logstash-input-elasticsearch (4.2.0)
logstash-input-exec (3.1.5)
logstash-input-file (4.0.3)
logstash-input-ganglia (3.1.3)
logstash-input-gelf (3.1.0)
logstash-input-generator (3.0.5)
logstash-input-graphite (3.0.4)
logstash-input-heartbeat (3.0.5)
logstash-input-http (3.0.8)
logstash-input-http_poller (3.3.4)
logstash-input-imap (3.0.5)
logstash-input-irc (3.0.6)
logstash-input-jdbc (4.3.3)
logstash-input-kafka (5.1.11)
logstash-input-log4j (3.1.2)
logstash-input-lumberjack (3.1.4)
logstash-input-pipe (3.0.6)
logstash-input-rabbitmq (5.2.5)
logstash-input-redis (3.1.6)
logstash-input-s3 (3.2.0)
logstash-input-snmptrap (3.0.5)
logstash-input-sqs (3.0.6)
logstash-input-stdin (3.2.5)
logstash-input-syslog (3.2.4)
logstash-input-tcp (4.2.4)
logstash-input-twitter (3.0.7)
logstash-input-udp (3.2.1)
logstash-input-unix (3.0.6)
logstash-input-xmpp (3.1.6)
logstash-output-cloudwatch (3.0.7)
logstash-output-csv (3.0.6)
logstash-output-elasticsearch (7.4.2)
logstash-output-file (4.2.1)
logstash-output-graphite (3.1.4)
logstash-output-http (4.4.0)
logstash-output-irc (3.0.5)
logstash-output-kafka (5.1.11)
logstash-output-nagios (3.0.5)
logstash-output-null (3.0.4)
logstash-output-pagerduty (3.0.6)
logstash-output-pipe (3.0.5)
logstash-output-rabbitmq (4.0.11)
logstash-output-redis (3.0.5)
logstash-output-s3 (4.0.13)
logstash-output-sns (4.0.6)
logstash-output-sqs (4.0.3)
logstash-output-statsd (3.1.4)
logstash-output-stdout (3.1.3)
logstash-output-tcp (4.0.2)
logstash-output-udp (3.0.5)
logstash-output-webhdfs (3.0.5)
logstash-output-xmpp (3.0.7)
logstash-patterns-core (4.1.2)

Used plugins:

# curl -s http://localhost:9600/_node/stats/pipeline | jq .pipeline.plugins | grep '"name":' | sort | uniq
      "name": "alter"
      "name": "beats",
      "name": "date"
      "name": "elasticsearch"
      "name": "geoip"
      "name": "grok"
      "name": "mutate"
      "name": "ruby"
      "name": "udp"
      "name": "useragent"

[javierE]

Hi, I'm facing a similar problem after upgrading to 5.6.8.
Found this issue on logstash-filter-grok , but has't upgrade/test it yet.

Issue: Suspected memory leak leading to heap OOM in Logstash 5.6.6

It seems to apply to your pluggin version .

[stefws (Steffen Winther Sørensen)]

Upgraded grok plugin from 4.0.2 to 4.0.3 and so far it seems better, ie. slow/more flat memory foot print.

[Faktu4noCaM]

check your logstash config! by default queue is set to memory and with memory leackage it fulls all heap and logstash goes down.

I`he 16Gb Ram only for Logstash. I start logstash, add one by one beats-clients (near 250 clients). it works fine for 3-4 weeks, but ram usage grows from 700 Mb to 16Gb and after that i have OOM error.
After restart, all 16Gb ram fulls in few minutes, because lots of client send a huge amount of metrics and logs, and ,by default logstash config, all of it puts to the ram.

After i change queue to persistant mode and set path for queue, i easily restart logstash without causing any errors. Only problem is that logstash need some time to process all queued info in the same time new metrics and logs puts to queue.

So, i think you need to set queue to persisted mode in logstash.yml in # ------------ Queuing Settings -------------- section

example

queue.type: persisted

path.queue: /home/queue/logstash

queue.max_bytes: 8gb

After that change i have 2 Gb for logstash. after start service it grows from 700Mb to 1900 and start to clean itself. 1900 Mb grows to 1964 Mb and returns to 1900 Mb, And it works in that way for 2 weeks till now without OOM error.

[stefws (Steffen Winther Sørensen)]

Got around 100 clients, but it seems grok plugin 4.0.3 fixed this issue for me.
Considering changing from grok to dissec at some point...

Drop in memory foot print @2018-03-21 00:00 was a restart due to OOM, then at 10:00 Iogstash got restarted w/grok 4.0.3 and memory growth seem more flat since

[Faktu4noCaM]

good for u! yesterday due to update 6.2.2 to 6.2.3 i`ve restart logstash. Memory has been growing from 623 Mb to 1020 Mb till now.