# LS 7.16.3 OutOfMemory jruby.RubyHash WatchedFilesCollection **URL:** \ **Category:** Logstash\ **Created:** [March 2, 2022, 2:24pm UTC](https://discuss.elastic.co/t/ls-7-16-3-outofmemory-jruby-rubyhash-watchedfilescollection/298629 "2022-03-02T14:24:13Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![ld\_pvl](https://avatars.discourse-cdn.com/v4/letter/l/cdc98d/32.png) [@ld\_pvl](https://discuss.elastic.co/u/ld_pvl)\ **Post date:** [March 2, 2022, 2:24pm UTC](https://discuss.elastic.co/t/ls-7-16-3-outofmemory-jruby-rubyhash-watchedfilescollection/298629/1 "2022-03-02T14:24:13Z")
My Logstash instances recently have started running OutOfMemory and need some help in identifying the cause. My logstash.conf has not been changed for a long time but I did do a few Logstash version upgrades in the past few months (though the upgrades were from 7.x to another 7.x) One notable attribute of my environment is that there are lot of log files and they also roll quite often so logstash has to keep track of a lot of "watched" files. Historically this hasn't been an issue until now when the OOMs started to happen. What values in logstash.conf should I tweak in relation to `org/logstash/filewatch/WatchedFilesCollection`? First option comes to mind is `max_open_files` but I'm not sure if there are any others I should try? ```auto file { sincedb_path => "some path" max_open_files => 10000 close_older => 0.001 stat_interval => 1 discover_interval => 5 sincedb_clean_after => 1 ignore_older => 259200 # 3 days path => "some path" type => "some type" start_position => "beginning" file_sort_direction => "desc" codec => multiline { patterns_dir => ["${LS_PATTERNS}"] pattern => "\[%{TIMESTAMP_ISO8601}" negate => "true" what => "previous" auto_flush_interval => 150 max_lines => 5000 ecs_compatibility => disabled } } ``` ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/6/6638ad2fad61d88172da9ed076255babd96c9965.png) ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 2, 2022, 6:23pm UTC](https://discuss.elastic.co/t/ls-7-16-3-outofmemory-jruby-rubyhash-watchedfilescollection/298629/2 "2022-03-02T18:23:46Z")
I have not verified that it would accumulate in that place, but if the pattern on a multiline codec does not match it will just keep accumulating data from the file waiting for it to match, and I believe that would happen down inside filewatch. You could try setting the [auto\_flush\_interval](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html#plugins-codecs-multiline-auto_flush_interval) option. If you think it will never take more than a minute to write out a complete error message then `auto_flush_interval => 60` should be OK. ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [March 3, 2022, 2:28pm UTC](https://discuss.elastic.co/t/ls-7-16-3-outofmemory-jruby-rubyhash-watchedfilescollection/298629/4 "2022-03-03T14:28:03Z")
Have you tried to increase memory sizes (Xms and Xmx1g) in jvm.options? ---
**Author:** ![ld\_pvl](https://avatars.discourse-cdn.com/v4/letter/l/cdc98d/32.png) [@ld\_pvl](https://discuss.elastic.co/u/ld_pvl)\ **Post date:** [March 3, 2022, 2:37pm UTC](https://discuss.elastic.co/t/ls-7-16-3-outofmemory-jruby-rubyhash-watchedfilescollection/298629/5 "2022-03-03T14:37:40Z")
Yep that is definitely an option but unfortunately for my case I cannot increase the Xmx values. ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [March 3, 2022, 3:25pm UTC](https://discuss.elastic.co/t/ls-7-16-3-outofmemory-jruby-rubyhash-watchedfilescollection/298629/6 "2022-03-03T15:25:15Z")
You have do almost everything on the input file settings. Have you tried to set pipeline params? pipeline.batch.size, pipeline.batch.delay, pipeline.workers? If delay is not a problem, maybe to split 2 or more pipelines, try with a persistent queue... Just ideas > **[Pipeline-to-pipeline communication | Logstash Reference \[8.0\] | Elastic](https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html)** > **[Best practices for Logstash](https://medium.com/ableneo/best-practices-for-logstash-81e1eb6a6262)** > > Most of us, working with elastic stack has come to the point where we have to make optimizations in matter of getting better indexing… ---
**Author:** ![ld\_pvl](https://avatars.discourse-cdn.com/v4/letter/l/cdc98d/32.png) [@ld\_pvl](https://discuss.elastic.co/u/ld_pvl)\ **Post date:** [March 3, 2022, 3:39pm UTC](https://discuss.elastic.co/t/ls-7-16-3-outofmemory-jruby-rubyhash-watchedfilescollection/298629/7 "2022-03-03T15:39:44Z")
The pipeline parameters suprisingly did not work for me. For testing purposes, I set batch size to 1 and number of workers to 1 and that did not have any positive impact whatsoever - my Logstash would still go OOM after some time. ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [March 3, 2022, 3:51pm UTC](https://discuss.elastic.co/t/ls-7-16-3-outofmemory-jruby-rubyhash-watchedfilescollection/298629/8 "2022-03-03T15:51:31Z")
I have almost same results for batch.size, batch.delay. Maybe I do something wrong, however not much impact. If you use the persistent queue, that means a message is on the disk, not in the memory. Maybe you can split 2-3 pipelines with heaver data and test. If you use codec =\> rubydebug, remove it. It consumes a lot of resources. ---
**Author:** ![ld\_pvl](https://avatars.discourse-cdn.com/v4/letter/l/cdc98d/32.png) [@ld\_pvl](https://discuss.elastic.co/u/ld_pvl)\ **Post date:** [March 3, 2022, 4:01pm UTC](https://discuss.elastic.co/t/ls-7-16-3-outofmemory-jruby-rubyhash-watchedfilescollection/298629/9 "2022-03-03T16:01:15Z")
Splitting into pipelines is a great idea to keep in mind for permitting future use cases. Unfortunately for me again, I have little control over the access to the environment so I can't really write to the disk. Also one difficulty I can see with the split of pipelines is it would depend on if we can reliably split between heavy and light data sets. In my case, for the same input, sometimes the client would "burst" a lot of heavy data spread across a lot of files, and some other times the same client would write very little data - so it is out of my control. The only option I have is to play around with the logstash settings to make sure it does not go over the Xmx limit even when there is a burst in data. ---
**Author:** ![ld\_pvl](https://avatars.discourse-cdn.com/v4/letter/l/cdc98d/32.png) [@ld\_pvl](https://discuss.elastic.co/u/ld_pvl)\ **Post date:** [March 3, 2022, 6:54pm UTC](https://discuss.elastic.co/t/ls-7-16-3-outofmemory-jruby-rubyhash-watchedfilescollection/298629/10 "2022-03-03T18:54:07Z")
From further testing, I found out that tweaking `max_open_files` and other settings suggested above actually do not provide any positive improvement and OOM still occurs. I found out the root cause to be an input plugin looking at a path that is something like this `/path/subdir/*/*/*.log` that returns a large number of log files, more than a million. When this input is disabled, by checking the garbage collection stats ([gcutil](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jstat.html)), I can see the memory pools look healthy. And when this input is enabled, the memory pools are maxed out straightaway: ```auto jstat -gcutil 23623 1000 S0 S1 E O M CCS YGC YGCT FGC FGCT CGC CGCT GCT 0.00 100.00 100.00 100.00 86.05 75.61 22 1.453 3 3.549 14 0.713 5.714 0.00 100.00 100.00 100.00 86.05 75.61 22 1.453 3 3.549 14 0.713 5.714 ``` So it looks like this is related to the `file discovery` process of the [file input plugin](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html). I checked the docs, there doesn't seem to be any option to reduce the list of files it keeps in memory during the discover process. If the list of files is really really large and Xmx is not big enough then Logstash just blows up. I think this calls for an "enhancement" for the file input plugin to not keep a big list - maybe just traverse in chunks of the list to make sure the memory is not overblown due to the large list of files Hi @ [guyboertje](https://discuss.elastic.co/u/guyboertje) 🙂, do you have any input on this? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 3, 2022, 7:00pm UTC](https://discuss.elastic.co/t/ls-7-16-3-outofmemory-jruby-rubyhash-watchedfilescollection/298629/11 "2022-03-03T19:00:25Z")
> [@ld\_pvl](#): > > Hi @ [guyboertje](https://discuss.elastic.co/u/guyboertje) 🙂, do you have any input on this? Guy is no longer working on the file input (or logstash). ---
**Author:** ![ld\_pvl](https://avatars.discourse-cdn.com/v4/letter/l/cdc98d/32.png) [@ld\_pvl](https://discuss.elastic.co/u/ld_pvl)\ **Post date:** [March 3, 2022, 7:14pm UTC](https://discuss.elastic.co/t/ls-7-16-3-outofmemory-jruby-rubyhash-watchedfilescollection/298629/12 "2022-03-03T19:14:43Z")
Thanks @Badger , I will open a ticket on github then with some steps to replicate the issue. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 31, 2022, 7:15pm UTC](https://discuss.elastic.co/t/ls-7-16-3-outofmemory-jruby-rubyhash-watchedfilescollection/298629/13 "2022-03-31T19:15:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.