Java heapspace issue while running logstash

Elastic Stack Logstash
1

Hi All,

I'm using logstash to ingest data into elasticsearch, while executing logstash its throwing up java heap space error.
There is almost 5450 documents i'm trying to ingest. Any ideas or suggestions where i should look into for this error?

/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/cronline.rb:77: warning: constant ::Fixnum is deprecated
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2048k  100 2048k    0     0   737k      0  0:00:02  0:00:02 --:--:--  737k
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid2962.hprof ...
Heap dump file created [2845468074 bytes in 2.787 secs]
warning: thread "[main]>worker2" terminated with exception (report_on_exception is true):
java.lang.OutOfMemoryError: Java heap space
        at org.jruby.RubyString.doClone(org/jruby/RubyString.java:1344)
        at org.logstash.Cloner.deep(org/logstash/Cloner.java:25)
        at org.logstash.Cloner.deepMap(org/logstash/Cloner.java:65)
        at org.logstash.Cloner.deep(org/logstash/Cloner.java:21)
        at org.logstash.Event.clone(org/logstash/Event.java:320)
        at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.clone(org/logstash/ext/JrubyEventExtLibrary.java:149)
        at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.clone(org/logstash/ext/JrubyEventExtLibrary.java:144)
        at java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(java/lang/invoke/DirectMethodHandle$Holder)
        at java.lang.invoke.LambdaForm$MH/0x00000008011cc440.invoke(java/lang/invoke/LambdaForm$MH)
        at java.lang.invoke.DelegatingMethodHandle$Holder.delegate(java/lang/invoke/DelegatingMethodHandle$Holder)
        at java.lang.invoke.LambdaForm$MH/0x00000008011c3040.guard(java/lang/invoke/LambdaForm$MH)
        at java.lang.invoke.DelegatingMethodHandle$Holder.delegate(java/lang/invoke/DelegatingMethodHandle$Holder)
        at java.lang.invoke.LambdaForm$MH/0x00000008011c3040.guard(java/lang/invoke/LambdaForm$MH)
        at java.lang.invoke.Invokers$Holder.linkToCallSite(java/lang/invoke/Invokers$Holder)
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_split_minus_3_dot_1_dot_8.lib.logstash.filters.split.filter(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-split-3.1.8/lib/logstash/filters/split.rb:89)
        at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java/lang/invoke/DirectMethodHandle$Holder)
        at java.lang.invoke.LambdaForm$MH/0x0000000801da9040.invoke(java/lang/invoke/LambdaForm$MH)
        at java.lang.invoke.LambdaForm$MH/0x0000000800bd6840.invokeExact_MT(java/lang/invoke/LambdaForm$MH)
        at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1814)
        at org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen)
        at java.lang.invoke.LambdaForm$DMH/0x0000000801171840.invokeVirtual(java/lang/invoke/LambdaForm$DMH)
        at java.lang.invoke.LambdaForm$MH/0x00000008011e0040.invoke(java/lang/invoke/LambdaForm$MH)
        at java.lang.invoke.Invokers$Holder.linkToCallSite(java/lang/invoke/Invokers$Holder)
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_split_minus_3_dot_1_dot_8.lib.logstash.filters.split.filter(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-split-3.1.8/lib/logstash/filters/split.rb:85)
[2020-05-27T18:30:09,611][ERROR][org.logstash.Logstash    ] java.lang.OutOfMemoryError: Java heap space

I'm using exec plugin to run curl and get data.

Thanks
Gautham

2

What does your filter configuration look like? Are you splitting an array?

3

@Badger i have 3 different filters configured,

filter
       {
         json { source => "message" }
         split { field => "results" }
       prune { blacklist_names => [ "command", "host", "message" ] }
}

This is my document,

"results": [
        {
            "IPAddress": "1.9.7.2",
            "Caption": "APPV1",
            "NodeID": 1,
            "DNS": "appv1.global.com",
            "Vendor": "Windows",
            "Location": "",
            "MachineType": "Windows 2016 Server",
            "StatusDescription": "Node status is Up, One or more application monitors is Down, Intel(R) 82574L Gigabit Network Connection #2 · eth2 is in an Unknown state, One or more AppInsight for IIS monitors is in an Unknown state.",
            "Status": 1,
            "Severity": 5001,
            "TotalMemory": 6.871887E+10,
            "PercentMemoryUsed": 30,
            "CPUCount": 25,
            "CPULoad": 28,
            "LastBoot": "2020-05-26T14:25:00.0000000",
            "StatusLED": "Up.gif              "
        },

If i try to ingest 2000 documents its working, when the count goes above 2000 then its throwing error.

Thanks
Gautham

4

Read this post. The rest of that thread provides background and explanation.

1 Like
5

@Badger Perfectly explained and it worked like a charm.. Thank you very much.

Thanks
Gautham

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.