# 3 questions about memory leak **URL:** \ **Category:** Logstash\ **Created:** [July 3, 2019, 3:38pm UTC](https://discuss.elastic.co/t/3-questions-about-memory-leak/188760 "2019-07-03T15:38:41Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![jcaballero](https://avatars.discourse-cdn.com/v4/letter/j/278dde/32.png) [@jcaballero](https://discuss.elastic.co/u/jcaballero)\ **Post date:** [July 3, 2019, 3:38pm UTC](https://discuss.elastic.co/t/3-questions-about-memory-leak/188760/1 "2019-07-03T15:38:41Z")
Using LogStash 6.7.1 It really feels like the memory consumption increases day by day. So: 1. Is really there a memory leak? I read in some of the "RELEASE NOTES" some comment about minimizing it, so I assume it's real. Has it been fixed completely in higher versions? 2. My pipelines use, a lot, the filter plugin "ruby". Could it be related? IIRC, the memory leak was related to JRuby... Is JRuby involved only when the filter plugin "ruby" is being used? 3. My setup is composed by 10 identical pipelines. Does that setup increases the effect of the memory leak? Would it be better if I reduce them to 5 pipelines, for example? Thanks a lot. Jose ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 5, 2019, 8:04am UTC](https://discuss.elastic.co/t/3-questions-about-memory-leak/188760/2 "2019-07-05T08:04:53Z")
Are you using the Monitoring functionality to track things? ---
**Author:** ![jcaballero](https://avatars.discourse-cdn.com/v4/letter/j/278dde/32.png) [@jcaballero](https://discuss.elastic.co/u/jcaballero)\ **Post date:** [July 5, 2019, 1:57pm UTC](https://discuss.elastic.co/t/3-questions-about-memory-leak/188760/3 "2019-07-05T13:57:21Z")
not really. A simple "ps" commands is enough to see how memory usage grows every day. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 5, 2019, 2:31pm UTC](https://discuss.elastic.co/t/3-questions-about-memory-leak/188760/4 "2019-07-05T14:31:23Z")
Is heap usage increasing over time or is it total memory reported used by the process? Are you using the file input plugin or persistent queues? ---
**Author:** ![jcaballero](https://avatars.discourse-cdn.com/v4/letter/j/278dde/32.png) [@jcaballero](https://discuss.elastic.co/u/jcaballero)\ **Post date:** [July 7, 2019, 7:10pm UTC](https://discuss.elastic.co/t/3-questions-about-memory-leak/188760/5 "2019-07-07T19:10:28Z")
I am checking RSS. Input plugin is "beats". ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 7, 2019, 7:30pm UTC](https://discuss.elastic.co/t/3-questions-about-memory-leak/188760/6 "2019-07-07T19:30:05Z")
Are you only using beats inputs? Have you got persistent queues configured? Do you have monitoring installed? ---
**Author:** ![jcaballero](https://avatars.discourse-cdn.com/v4/letter/j/278dde/32.png) [@jcaballero](https://discuss.elastic.co/u/jcaballero)\ **Post date:** [July 9, 2019, 6:18pm UTC](https://discuss.elastic.co/t/3-questions-about-memory-leak/188760/7 "2019-07-09T18:18:03Z")
Nope, I don't have persistent queues. I admit I didn't know about them until now 🙂 No monitoring. ---
**Author:** ![jcaballero](https://avatars.discourse-cdn.com/v4/letter/j/278dde/32.png) [@jcaballero](https://discuss.elastic.co/u/jcaballero)\ **Post date:** [July 16, 2019, 8:02pm UTC](https://discuss.elastic.co/t/3-questions-about-memory-leak/188760/9 "2019-07-16T20:02:49Z")
unclear to me if any of the comments help me to answer my original questions... ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 17, 2019, 4:06am UTC](https://discuss.elastic.co/t/3-questions-about-memory-leak/188760/10 "2019-07-17T04:06:03Z")
Can you show your config or list the plugins you are using? If you have any plugins that rely on files being read the reported memory usage could be growing due to files being cached by the OS, which is not a memory leak. If that was the case i would expect reported memory use to go down once Logstash is restarted. ---
**Author:** ![jcaballero](https://avatars.discourse-cdn.com/v4/letter/j/278dde/32.png) [@jcaballero](https://discuss.elastic.co/u/jcaballero)\ **Post date:** [July 17, 2019, 3:01pm UTC](https://discuss.elastic.co/t/3-questions-about-memory-leak/188760/11 "2019-07-17T15:01:57Z")
Sure. Here it comes. It is a mock, for privacy. The key aspect is that I use the Ruby filter plugin, which keeps in memory several Hashes, where each key value is a combination of hostname and logfile for the source input data. That is a finite number, so at some point it should have created all possible values. I also guess those Hashes are eventually recycled when the value of the key is repeated, as they are class variables, correct? Ruby is not keeping all of them in memory forever, right? ``` input { beats { } } # ========================================================================= filter { if [fields][log_type] == "foo" { mutate { split => ["source", "/"] add_field => { "logfile" => "%{[source][-1]}" } add_field => { "origin" => "%{logfile}@%{[beat][hostname]}" } } grok { pattern_definitions => { ... } match => { "message" => [ ... ] } break_on_match => false } date { match => ["eventtimestamp", "MM/dd/yy HH:mm:ss.SSS"] target => "eventtimestamp" } ruby { init => ' @var1 = Hash.new @var2 = Hash.new @var3 = Hash.new @var4 = Hash.new @var5 = Hash.new @var6 = Hash.new ... ' code => ' @origin = event.get("origin") if .... @var1[@origin] = event.get("var1") elsif ... @var2[@origin] = event.get("var2") elsif ... @var3[@origin] = event.get("var3") elseif ... event.set("var1", @var1[@origin]) event.set("var2", @var2[@origin]) event.set("var3", @var3[@origin]) ... else event.cancel end ' } mutate { add_field => { "executionhost" => "%{[host][name]}" } } prune { whitelist_names => ["...", "..."] } } else if [fields][log_type] == "bar" { # # similar workflow # } } # ========================================================================= output { lumberjack { codec => json hosts => "myremotehost" ssl_certificate => "/etc/logstash/certs/lumberjack.cert" port => 1234 } } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 14, 2019, 3:01pm UTC](https://discuss.elastic.co/t/3-questions-about-memory-leak/188760/12 "2019-08-14T15:01:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.