Logstash 2.1 UDP Listener Died

We are using a single logstash instance which runs on c4.xlarge instance in AWS.
The input plugin is as following:

udp {
port => 7192
workers => 4
queue_size => 40000
codec => json {
charset => "UTF-8"

We are seeing in the logs the following error:
{:timestamp=>"2017-02-23T09:13:03.992000+0000", :message=>"UDP listener died", :exception=>#<IO::EAGAINWaitReadable: Resource temporarily unavailable - errno backtraces disabled; run with -Xerrno.backtrace=true to enable>, :backtrace=>[], :level=>:warn}

When I try to add -Xerrno.backtrace=true to the logstash server it will not start up and I get this error:
Unrecognized option: -Xerrno.backtrace=true

Additionally, when I check the port via netstat I see that sometimes logstash really not opening the port, which causes messages disappear.

To try and fix this issue I have added another logstash server that resolves via consul DNS (causes round robin between the 2 logstash instances). The error messages seemed to decrease and the amount of indexed objects to Elasticsearch increased by 2, but over time the message began to show on both logstash instances. I found also the following error couple of days later:
{:timestamp=>"2017-02-26T06:45:25.246000+0000", :message=>"UDP listener died", :exception=>#<ThreadError: unable to create new native thread>, :backtrace=>["org/jruby/RubyThread.java:441:in initialize'", "org/jruby/RubyThread.java:390:in new'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.3/lib/logstash/inputs/udp.rb:73:in udp_listener'", "org/jruby/RubyF ixnum.java:280:in times'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.3/lib/logstash/inputs/udp.rb:71:in udp_listener'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/ logstash-input-udp-2.0.3/lib/logstash/inputs/udp.rb:50:in run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.3-java/lib/logstash/pipeline.rb:206:in inputworker'", "/opt/lo gstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.3-java/lib/logstash/pipeline.rb:199:in start_input'"], :level=>:warn}

Is it load that causes Logstash to drop it`s UDP connections? So adding more than 2 machines will help?
Or is it something else entirely that can cause this thing?


Update: I started 2 more logstash instances and they all get the same "UDP listener died" errors.

Can someone help me to debug it or know what can cause this?


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.