java.lang.OutOfMemoryError: Java heap space in Logstash

Elastic Stack Logstash
1

I have too many configuration files. I run these files together via pipelines.yml. I use these types of settings likewise below;

- pipeline.id: pipeline1
  path.config: "path_to_pipeline1"
- pipeline.id: pipeline2
  path.config: "path_to_pipeline2"
- pipeline.id: pipeline3
  path.config: "path_to_pipeline3"
  .....

I set JVM HeapSize as 10gb with the configuration below; (I have also tried 8gb and I have 16gb ram)

-Xms10g
-Xmx10g

If I run my pipelines one by one. It works perfectly but when I try to run multiple pipelines after some hours I get this error

java.lang.OutOfMemoryError: Java heap space
        at org.jruby.util.io.OpenFile.binwrite(OpenFile.java:2165)
        at org.jruby.util.io.OpenFile.fwrite(OpenFile.java:2076)
        at org.jruby.RubyIO.write(RubyIO.java:1474)
        at org.jruby.RubyIO.write(RubyIO.java:1437)
        at org.jruby.RubyIO$INVOKER$i$write.call(RubyIO$INVOKER$i$write.gen)
        at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:819)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:203)
        at org.jruby.RubyIO.write(RubyIO.java:2638)
        at org.jruby.RubyIO.putsSingle(RubyIO.java:2589)
        at org.jruby.RubyIO.puts1(RubyIO.java:2508)
        at org.jruby.RubyIO.puts(RubyIO.java:2481)
        at org.jruby.RubyIO$INVOKER$i$puts.call(RubyIO$INVOKER$i$puts.gen)
        at java.lang.invoke.LambdaForm$DMH/1974701110.invokeVirtual_L6_L(LambdaForm$DMH)
        at java.lang.invoke.LambdaForm$BMH/2145760212.reinvoke(LambdaForm$BMH)
        at java.lang.invoke.LambdaForm$MH/2001112025.delegate(LambdaForm$MH)
        at java.lang.invoke.LambdaForm$MH/757819744.guard(LambdaForm$MH)
        at java.lang.invoke.LambdaForm$MH/2001112025.delegate(LambdaForm$MH)
        at java.lang.invoke.LambdaForm$MH/757819744.guard(LambdaForm$MH)
        at java.lang.invoke.LambdaForm$MH/1375297293.linkToCallSite(LambdaForm$MH)
        at home.logstash.plugins.logstash_minus_codec_minus_my_plugin.lib.logstash.codecs.my_plugin.RUBY$method$file_write$0(/home/logstash/plugins/logstash-codec-my_plugin/lib/logstash/codecs/my_plugin.rb:95)
        at java.lang.invoke.LambdaForm$DMH/369219075.invokeStatic_L7_L(LambdaForm$DMH)
        at java.lang.invoke.LambdaForm$MH/291284958.invokeExact_MT(LambdaForm$MH)
        at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:106)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:140)
        at org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:278)
        at java.lang.invoke.LambdaForm$DMH/1974701110.invokeVirtual_L6_L(LambdaForm$DMH)
        at java.lang.invoke.LambdaForm$BMH/592048461.reinvoke(LambdaForm$BMH)
        at java.lang.invoke.LambdaForm$MH/1375297293.linkToCallSite(LambdaForm$MH)
        at home.logstash.plugins.logstash_minus_codec_minus_my_plugin.lib.logstash.codecs.my_plugin.RUBY$method$file_write$0(/home/logstash/plugins/logstash-codec-my_plugin/lib/logstash/codecs/my_plugin.rb:95)
        at java.lang.invoke.LambdaForm$DMH/75254713.invokeStatic_L7_L(LambdaForm$DMH)
        at java.lang.invoke.LambdaForm$MH/291284958.invokeExact_MT(LambdaForm$MH)

In my codec plugin I've just use simple file write in ruby

def file_write(lorem_data)
    write_to_file = File.open(@location, "w")
    write_to_file.puts lorem_data.to_s
    write_to_file.close
  end

Thanks for answering.

2

Let's keep this to a single topic as it makes it easier to help you, plus there's a lot of existing info in your original topic java.lang.OutOfMemoryError: Java heap space