Hi All,
I have been trying to ingest data which is of 6000 documents, its around 73 MB, but i'm repeatedly getting java heap space issue.
below is the error.
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid6498.hprof ...
Heap dump file created [4360270050 bytes in 6.815 secs]
warning: thread "[main]>worker3" terminated with exception (report_on_exception is true):
java.lang.OutOfMemoryError: Java heap space
at org.jruby.RubyString.doClone(org/jruby/RubyString.java:1344)
at org.logstash.Cloner.deep(org/logstash/Cloner.java:25)
at org.logstash.Cloner.deepMap(org/logstash/Cloner.java:65)
at org.logstash.Cloner.deep(org/logstash/Cloner.java:21)
at org.logstash.Event.clone(org/logstash/Event.java:320)
at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.clone(org/logstash/ext/JrubyEventExtLibrary.java:149)
at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.clone(org/logstash/ext/JrubyEventExtLibrary.java:144)
at java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(java/lang/invoke/DirectMethodHandle$Holder)
at java.lang.invoke.LambdaForm$MH/0x00000008011cc440.invoke(java/lang/invoke/LambdaForm$MH)
at java.lang.invoke.DelegatingMethodHandle$Holder.delegate(java/lang/invoke/DelegatingMethodHandle$Holder)
at java.lang.invoke.LambdaForm$MH/0x00000008011c3040.guard(java/lang/invoke/LambdaForm$MH)
at java.lang.invoke.DelegatingMethodHandle$Holder.delegate(java/lang/invoke/DelegatingMethodHandle$Holder)
at java.lang.invoke.LambdaForm$MH/0x00000008011c3040.guard(java/lang/invoke/LambdaForm$MH)
at java.lang.invoke.Invokers$Holder.linkToCallSite(java/lang/invoke/Invokers$Holder)
at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_split_minus_3_dot_1_dot_8.lib.logstash.filters.split.filter(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-split-3.1.8/lib/logstash/filters/split.rb:89)
at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java/lang/invoke/DirectMethodHandle$Holder)
at java.lang.invoke.LambdaForm$MH/0x00000008011df440.invoke(java/lang/invoke/LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/0x0000000800bd6840.invokeExact_MT(java/lang/invoke/LambdaForm$MH)
at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1814)
at org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen)
at java.lang.invoke.LambdaForm$DMH/0x0000000801171840.invokeVirtual(java/lang/invoke/LambdaForm$DMH)
at java.lang.invoke.LambdaForm$MH/0x00000008011e0040.invoke(java/lang/invoke/LambdaForm$MH)
at java.lang.invoke.Invokers$Holder.linkToCallSite(java/lang/invoke/Invokers$Holder)
at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_split_minus_3_dot_1_dot_8.lib.logstash.filters.split.filter(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-split-3.1.8/lib/logstash/filters/split.rb:85)
[ERROR] 2020-08-02 13:04:51.142 [LogStash::Runner] Logstash - java.lang.OutOfMemoryError: Java heap space
I my config file below is the only filter i'm using
filter
{
json { source => "message" }
split { field => "result" }
prune { blacklist_names => [ "command", "host", "message" ] }
}
Any advice on this please.
Thanks
Gautham