# 5.6.8 java.lang.OutOfMemoryError after 6-10 days **URL:** \ **Category:** Logstash\ **Created:** [March 16, 2018, 10:21am UTC](https://discuss.elastic.co/t/5-6-8-java-lang-outofmemoryerror-after-6-10-days/124265 "2018-03-16T10:21:21Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![stefws](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stefws/32/6442_2.png) [@stefws](https://discuss.elastic.co/u/stefws)\ **Post date:** [March 16, 2018, 10:21am UTC](https://discuss.elastic.co/t/5-6-8-java-lang-outofmemoryerror-after-6-10-days/124265/1 "2018-03-16T10:21:21Z")
Since we patched from 5.6.5 to 5.6.8 we're seeing logstash memory leakage so it'll now run out of memory after 6-10 days, nothing changed in our filters etc. So assuming there's a leak somewhere maybe. Before memory usage was steady (see memory trend graph). Anyone else seeing this or know of a possible issue/workaround/fix, please let me know. TIA ``` # rpm -qi logstash Name : logstash Relocations: / Version : 5.6.8 Vendor: Elasticsearch Release : 1 Build Date: Fri 16 Feb 2018 07:11:44 PM CET Install Date: Thu 22 Feb 2018 12:26:39 PM CET Build Host: packer-virtualbox-iso-1518356860 ``` ... ![28](https://us1.discourse-cdn.com/elastic/original/3X/8/2/826168ae8e7b0ca04390478f7d10527668397fb7.png) ---
**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\ **Post date:** [March 17, 2018, 4:47am UTC](https://discuss.elastic.co/t/5-6-8-java-lang-outofmemoryerror-after-6-10-days/124265/2 "2018-03-17T04:47:38Z")
Any info you're willing to provide about how your pipelines are configured, including which plugins they use (and what versions), heap settings, how many workers, whether you've got persistent queueing enabled, etc, will be helpful in narrowing down the surface area of the problem. ---
**Author:** ![stefws](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stefws/32/6442_2.png) [@stefws](https://discuss.elastic.co/u/stefws)\ **Post date:** [March 20, 2018, 8:30am UTC](https://discuss.elastic.co/t/5-6-8-java-lang-outofmemoryerror-after-6-10-days/124265/3 "2018-03-20T08:30:21Z")
Whatever you desire... pipelines are processing req.s from filebeat/metricbeats also now 5.6.8 and collectd. JVM options: ``` # awk '!/^#/ && NF!=0{print}' jvm.options -Xms512m -Xmx4g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError ``` Logstash config: ``` # awk '!/^#/ && NF!=0{print}' logstash.yml path.data: /var/lib/logstash pipeline.workers: 8 pipeline.output.workers: 2 pipeline.batch.size: 125 pipeline.batch.delay: 5 path.config: /etc/logstash/conf.d log.level: info path.logs: /var/log/logstash ``` Installed plugins (not all used): ``` # cd /usr/share/logstash; bin/logstash-plugin list --verbose logstash-codec-cef (4.1.4) logstash-codec-collectd (3.0.8) logstash-codec-dots (3.0.6) logstash-codec-edn (3.0.6) logstash-codec-edn_lines (3.0.6) logstash-codec-es_bulk (3.0.6) logstash-codec-fluent (3.1.5) logstash-codec-graphite (3.0.5) logstash-codec-json (3.0.5) logstash-codec-json_lines (3.0.5) logstash-codec-line (3.0.8) logstash-codec-msgpack (3.0.7) logstash-codec-multiline (3.0.9) logstash-codec-netflow (3.11.0) logstash-codec-plain (3.0.6) logstash-codec-rubydebug (3.0.5) logstash-filter-alter (3.0.2) logstash-filter-cidr (3.1.2) logstash-filter-clone (3.0.5) logstash-filter-csv (3.0.8) logstash-filter-date (3.1.9) logstash-filter-dissect (1.1.4) logstash-filter-dns (3.0.7) logstash-filter-drop (3.0.5) logstash-filter-fingerprint (3.1.2) logstash-filter-geoip (4.3.1) logstash-filter-grok (4.0.2) logstash-filter-json (3.0.5) logstash-filter-kv (4.0.3) logstash-filter-metrics (4.0.5) logstash-filter-mutate (3.2.0) logstash-filter-ruby (3.1.3) logstash-filter-sleep (3.0.6) logstash-filter-split (3.1.6) logstash-filter-syslog_pri (3.0.5) logstash-filter-throttle (4.0.4) logstash-filter-translate (3.0.4) logstash-filter-urldecode (3.0.6) logstash-filter-useragent (3.2.2) logstash-filter-uuid (3.0.5) logstash-filter-xml (4.0.5) logstash-input-beats (3.1.26) logstash-input-couchdb_changes (3.1.4) logstash-input-dead_letter_queue (1.1.2) logstash-input-elasticsearch (4.2.0) logstash-input-exec (3.1.5) logstash-input-file (4.0.3) logstash-input-ganglia (3.1.3) logstash-input-gelf (3.1.0) logstash-input-generator (3.0.5) logstash-input-graphite (3.0.4) logstash-input-heartbeat (3.0.5) logstash-input-http (3.0.8) logstash-input-http_poller (3.3.4) logstash-input-imap (3.0.5) logstash-input-irc (3.0.6) logstash-input-jdbc (4.3.3) logstash-input-kafka (5.1.11) logstash-input-log4j (3.1.2) logstash-input-lumberjack (3.1.4) logstash-input-pipe (3.0.6) logstash-input-rabbitmq (5.2.5) logstash-input-redis (3.1.6) logstash-input-s3 (3.2.0) logstash-input-snmptrap (3.0.5) logstash-input-sqs (3.0.6) logstash-input-stdin (3.2.5) logstash-input-syslog (3.2.4) logstash-input-tcp (4.2.4) logstash-input-twitter (3.0.7) logstash-input-udp (3.2.1) logstash-input-unix (3.0.6) logstash-input-xmpp (3.1.6) logstash-output-cloudwatch (3.0.7) logstash-output-csv (3.0.6) logstash-output-elasticsearch (7.4.2) logstash-output-file (4.2.1) logstash-output-graphite (3.1.4) logstash-output-http (4.4.0) logstash-output-irc (3.0.5) logstash-output-kafka (5.1.11) logstash-output-nagios (3.0.5) logstash-output-null (3.0.4) logstash-output-pagerduty (3.0.6) logstash-output-pipe (3.0.5) logstash-output-rabbitmq (4.0.11) logstash-output-redis (3.0.5) logstash-output-s3 (4.0.13) logstash-output-sns (4.0.6) logstash-output-sqs (4.0.3) logstash-output-statsd (3.1.4) logstash-output-stdout (3.1.3) logstash-output-tcp (4.0.2) logstash-output-udp (3.0.5) logstash-output-webhdfs (3.0.5) logstash-output-xmpp (3.0.7) logstash-patterns-core (4.1.2) ``` Used plugins: ``` # curl -s http://localhost:9600/_node/stats/pipeline | jq .pipeline.plugins | grep '"name":' | sort | uniq "name": "alter" "name": "beats", "name": "date" "name": "elasticsearch" "name": "geoip" "name": "grok" "name": "mutate" "name": "ruby" "name": "udp" "name": "useragent" ``` ---
**Author:** ![javierE](https://avatars.discourse-cdn.com/v4/letter/j/cc9497/32.png) [@javierE](https://discuss.elastic.co/u/javierE)\ **Post date:** [March 20, 2018, 12:19pm UTC](https://discuss.elastic.co/t/5-6-8-java-lang-outofmemoryerror-after-6-10-days/124265/4 "2018-03-20T12:19:24Z")
Hi, I'm facing a similar problem after upgrading to 5.6.8. Found this issue on logstash-filter-grok , but has't upgrade/test it yet. > It seems to apply to your pluggin version . ---
**Author:** ![stefws](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stefws/32/6442_2.png) [@stefws](https://discuss.elastic.co/u/stefws)\ **Post date:** [March 21, 2018, 12:59pm UTC](https://discuss.elastic.co/t/5-6-8-java-lang-outofmemoryerror-after-6-10-days/124265/5 "2018-03-21T12:59:51Z")
Upgraded grok plugin from 4.0.2 to 4.0.3 and so far it seems better, ie. slow/more flat memory foot print. ![16](https://us1.discourse-cdn.com/elastic/original/3X/d/2/d2dc716e3323a3d5091fd300844ee5cbaedd0c1a.png) ---
**Author:** ![Faktu4noCaM](https://avatars.discourse-cdn.com/v4/letter/f/71c47a/32.png) [@Faktu4noCaM](https://discuss.elastic.co/u/Faktu4noCaM)\ **Post date:** [March 22, 2018, 2:53pm UTC](https://discuss.elastic.co/t/5-6-8-java-lang-outofmemoryerror-after-6-10-days/124265/6 "2018-03-22T14:53:16Z")
check your logstash config! by default queue is set to memory and with memory leackage it fulls all heap and logstash goes down. I`he 16Gb Ram only for Logstash. I start logstash, add one by one beats-clients (near 250 clients). it works fine for 3-4 weeks, but ram usage grows from 700 Mb to 16Gb and after that i have OOM error. After restart, all 16Gb ram fulls in few minutes, because lots of client send a huge amount of metrics and logs, and ,by default logstash config, all of it puts to the ram. After i change queue to persistant mode and set path for queue, i easily restart logstash without causing any errors. Only problem is that logstash need some time to process all queued info in the same time new metrics and logs puts to queue. So, i think you need to set queue to persisted mode in logstash.yml in # ------------ Queuing Settings -------------- section > **example** > > queue.type: persisted > > path.queue: /home/queue/logstash > > queue.max\_bytes: 8gb After that change i have 2 Gb for logstash. after start service it grows from 700Mb to 1900 and start to clean itself. 1900 Mb grows to 1964 Mb and returns to 1900 Mb, And it works in that way for 2 weeks till now without OOM error. ---
**Author:** ![stefws](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stefws/32/6442_2.png) [@stefws](https://discuss.elastic.co/u/stefws)\ **Post date:** [March 23, 2018, 9:41am UTC](https://discuss.elastic.co/t/5-6-8-java-lang-outofmemoryerror-after-6-10-days/124265/7 "2018-03-23T09:41:27Z")
Got around 100 clients, but it seems grok plugin 4.0.3 fixed this issue for me. Considering changing from grok to dissec at some point... Drop in memory foot print @2018-03-21 00:00 was a restart due to OOM, then at 10:00 Iogstash got restarted w/grok 4.0.3 and memory growth seem more flat since 🙂 ![33](https://us1.discourse-cdn.com/elastic/original/3X/8/b/8bae0b27e086f9b9ae06f8c6901e4f36d82600f9.png) ---
**Author:** ![Faktu4noCaM](https://avatars.discourse-cdn.com/v4/letter/f/71c47a/32.png) [@Faktu4noCaM](https://discuss.elastic.co/u/Faktu4noCaM)\ **Post date:** [March 23, 2018, 10:37am UTC](https://discuss.elastic.co/t/5-6-8-java-lang-outofmemoryerror-after-6-10-days/124265/8 "2018-03-23T10:37:48Z")
good for u! yesterday due to update 6.2.2 to 6.2.3 i`ve restart logstash. Memory has been growing from 623 Mb to 1020 Mb till now. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 20, 2018, 10:37am UTC](https://discuss.elastic.co/t/5-6-8-java-lang-outofmemoryerror-after-6-10-days/124265/9 "2018-04-20T10:37:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.