Java heap space issue with logstash

[Gauti (Gautham)]

Hi All,

I have been trying to ingest data which is of 6000 documents, its around 73 MB, but i'm repeatedly getting java heap space issue.

below is the error.

java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid6498.hprof ...
Heap dump file created [4360270050 bytes in 6.815 secs]
warning: thread "[main]>worker3" terminated with exception (report_on_exception is true):
java.lang.OutOfMemoryError: Java heap space
        at org.jruby.RubyString.doClone(org/jruby/RubyString.java:1344)
        at org.logstash.Cloner.deep(org/logstash/Cloner.java:25)
        at org.logstash.Cloner.deepMap(org/logstash/Cloner.java:65)
        at org.logstash.Cloner.deep(org/logstash/Cloner.java:21)
        at org.logstash.Event.clone(org/logstash/Event.java:320)
        at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.clone(org/logstash/ext/JrubyEventExtLibrary.java:149)
        at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.clone(org/logstash/ext/JrubyEventExtLibrary.java:144)
        at java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(java/lang/invoke/DirectMethodHandle$Holder)
        at java.lang.invoke.LambdaForm$MH/0x00000008011cc440.invoke(java/lang/invoke/LambdaForm$MH)
        at java.lang.invoke.DelegatingMethodHandle$Holder.delegate(java/lang/invoke/DelegatingMethodHandle$Holder)
        at java.lang.invoke.LambdaForm$MH/0x00000008011c3040.guard(java/lang/invoke/LambdaForm$MH)
        at java.lang.invoke.DelegatingMethodHandle$Holder.delegate(java/lang/invoke/DelegatingMethodHandle$Holder)
        at java.lang.invoke.LambdaForm$MH/0x00000008011c3040.guard(java/lang/invoke/LambdaForm$MH)
        at java.lang.invoke.Invokers$Holder.linkToCallSite(java/lang/invoke/Invokers$Holder)
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_split_minus_3_dot_1_dot_8.lib.logstash.filters.split.filter(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-split-3.1.8/lib/logstash/filters/split.rb:89)
        at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java/lang/invoke/DirectMethodHandle$Holder)
        at java.lang.invoke.LambdaForm$MH/0x00000008011df440.invoke(java/lang/invoke/LambdaForm$MH)
        at java.lang.invoke.LambdaForm$MH/0x0000000800bd6840.invokeExact_MT(java/lang/invoke/LambdaForm$MH)
        at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1814)
        at org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen)
        at java.lang.invoke.LambdaForm$DMH/0x0000000801171840.invokeVirtual(java/lang/invoke/LambdaForm$DMH)
        at java.lang.invoke.LambdaForm$MH/0x00000008011e0040.invoke(java/lang/invoke/LambdaForm$MH)
        at java.lang.invoke.Invokers$Holder.linkToCallSite(java/lang/invoke/Invokers$Holder)
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_split_minus_3_dot_1_dot_8.lib.logstash.filters.split.filter(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-split-3.1.8/lib/logstash/filters/split.rb:85)
[ERROR] 2020-08-02 13:04:51.142 [LogStash::Runner] Logstash - java.lang.OutOfMemoryError: Java heap space

I my config file below is the only filter i'm using

filter
       {
        json { source => "message" }
        split { field => "result" }
        prune { blacklist_names => [ "command", "host", "message" ] }
    }

Any advice on this please.

Thanks
Gautham

[Badger]

You could try moving the prune ahead of the split, but I doubt that will make a lot of difference.

Get a heap dump analyzer (like MAT) and see what the leak suspects in the dump are.