OOM ElasticSearch OUTPUT

Elastic Stack Logstash
1

Using LogStash version 6.2.4.
I have a rather complex pipeline that crashes when I use elasticsearch{}output, but works fine when I used stdout { codec => rubydebug}.

There are around 115k document - also quite simple - being outputted, almost no free text, mostly attribute values.

I watched `/_node/stats/event and I saw it just have out: 0, all of a sudden crash. Not that it ramped up or something, just immediately crashed.

I tired running with -b 250, -b 125, b -75

i tried setting -Xms and -Xmx to 3g and it didn't crash. but I'm not sure this should crash becasue there's too much data. Can I throttle this plugin? change batch size perhaps or workers. I'm lost.

The output configuration:

elasticsearch {
            hosts => "localhost:9200"
            user => "<USER>"
            password => "<PASSWORD>"

            action => update
            index => <index>
            document_id => "%{id}"

            scripted_upsert => true
            script => '
           ctx._source["dummy"] = "dummy" // Just so the script will be compiled
            '
        }

I don't get much logs, but here's what I have

java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid30485.hprof ...
Heap dump file created [1499118783 bytes in 9.915 secs]
2018-05-16 16:21:34 +0300: Listen loop error: #<IOError: closed stream>
org/jruby/RubyIO.java:3405:in `select'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:322:in `handle_servers'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:296:in `block in run'
2018-05-16 16:21:34 +0300: Listen loop error: #<IOError: closed stream>
org/jruby/RubyIO.java:3405:in `select'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:322:in `handle_servers'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:296:in `block in run'
2018-05-16 16:21:34 +0300: Listen loop error: #<IOError: closed stream>
org/jruby/RubyIO.java:3405:in `select'

I tried profiling with VisualVM:
Heap space
CPU
Memory sampling

2

up up up up

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.