Logstash runs out of memory as soon pipelines are triggered

Elastic Stack Logstash
1

My logstash has started suddenly running out of memory, though no changes have been made to the pipeline. Following is the error log.

My server has 14-15 GB of memory free. Tried allocating different memory in jvm.options : 512m, 1g,2g and 4g . Nothing works.

[90.242s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[90.243s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[90.244s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[90.246s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[90.247s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[90.248s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[90.250s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[2022-06-09T17:44:47,698][INFO ][logstash.agent           ] Pipelines running {:count=>8, :running_pipelines=>[:user_simulations_csv, :user_simulations_2_csv, :gsmOfferDump, :mom_offer_csv, :project_2_csv, :project_activity_2_csv, :project_csv, :project_activity_csv], :non_running_pipelines=>[]}
[2022-06-09T17:44:48,712][INFO ][logstash.javapipeline    ][user_simulations_2_csv] Pipeline terminated {"pipeline.id"=>"user_simulations_2_csv"}
[2022-06-09T17:44:48,712][INFO ][logstash.javapipeline    ][project_activity_2_csv] Pipeline terminated {"pipeline.id"=>"project_activity_2_csv"}
[2022-06-09T17:44:48,713][INFO ][logstash.javapipeline    ][gsmOfferDump] Pipeline terminated {"pipeline.id"=>"gsmOfferDump"}
[2022-06-09T17:44:48,831][INFO ][logstash.javapipeline    ][user_simulations_csv] Pipeline terminated {"pipeline.id"=>"user_simulations_csv"}
[2022-06-09T17:44:48,832][INFO ][logstash.javapipeline    ][mom_offer_csv] Pipeline terminated {"pipeline.id"=>"mom_offer_csv"}
[2022-06-09T17:44:48,856][INFO ][logstash.javapipeline    ][project_activity_csv] Pipeline terminated {"pipeline.id"=>"project_activity_csv"}
[2022-06-09T17:44:48,856][INFO ][logstash.javapipeline    ][project_2_csv] Pipeline terminated {"pipeline.id"=>"project_2_csv"}
[2022-06-09T17:44:48,856][INFO ][logstash.javapipeline    ][project_csv] Pipeline terminated {"pipeline.id"=>"project_csv"}
[2022-06-09T17:44:48,878][FATAL][org.logstash.Logstash    ]
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
        at java.lang.Thread.start0(Native Method) ~[?:?]
        at java.lang.Thread.start(Thread.java:798) ~[?:?]
        at org.jruby.internal.runtime.NativeThread.start(NativeThread.java:54) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyThread.startThread(RubyThread.java:626) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyThread.initialize(RubyThread.java:605) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyThread.startThread(RubyThread.java:696) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyThread.start(RubyThread.java:567) ~[jruby-complete-9.2.20.1.jar:?]
        at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.thwait.RUBY$block$join_nowait$1(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/thwait.rb:91) ~[?:?]
        at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:148) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.runtime.BlockBody.yield(BlockBody.java:106) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.runtime.Block.yield(Block.java:184) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyArray.each(RubyArray.java:1821) ~[jruby-complete-9.2.20.1.jar:?]
        at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.thwait.RUBY$method$join_nowait$0(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/thwait.rb:90) ~[?:?]
        at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.thwait.RUBY$method$initialize$0(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/thwait.rb:54) ~[?:?]

Total allocated threads to my user are : 4096. But I don't know how can I find threads consumed by my user. Though I don't assume these many have been used.

Or is there something that is causing this issue, Can someone help me figure this out.

2

This is not logstash specific. Java is not running out of heap, so increasing the heap size will not help and can make things worse by decreasing the available native memory. This page has some suggestions, otherwise Google the error message.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.