Hi,
We are using a single logstash instance which runs on c4.xlarge instance in AWS.
The input plugin is as following:
udp {
port => 7192
workers => 4
queue_size => 40000
codec => json {
charset => "UTF-8"
}
We are seeing in the logs the following error:
{:timestamp=>"2017-02-23T09:13:03.992000+0000", :message=>"UDP listener died", :exception=>#<IO::EAGAINWaitReadable: Resource temporarily unavailable - errno backtraces disabled; run with -Xerrno.backtrace=true to enable>, :backtrace=>[], :level=>:warn}
When I try to add -Xerrno.backtrace=true to the logstash server it will not start up and I get this error:
Unrecognized option: -Xerrno.backtrace=true
Additionally, when I check the port via netstat I see that sometimes logstash really not opening the port, which causes messages disappear.
To try and fix this issue I have added another logstash server that resolves via consul DNS (causes round robin between the 2 logstash instances). The error messages seemed to decrease and the amount of indexed objects to Elasticsearch increased by 2, but over time the message began to show on both logstash instances. I found also the following error couple of days later:
{:timestamp=>"2017-02-26T06:45:25.246000+0000", :message=>"UDP listener died", :exception=>#<ThreadError: unable to create new native thread>, :backtrace=>["org/jruby/RubyThread.java:441:in initialize'", "org/jruby/RubyThread.java:390:in new'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.3/lib/logstash/inputs/udp.rb:73:in udp_listener'", "org/jruby/RubyF ixnum.java:280:in times'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.3/lib/logstash/inputs/udp.rb:71:in udp_listener'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/ logstash-input-udp-2.0.3/lib/logstash/inputs/udp.rb:50:in run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.3-java/lib/logstash/pipeline.rb:206:in inputworker'", "/opt/lo gstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.3-java/lib/logstash/pipeline.rb:199:in start_input'"], :level=>:warn}
Is it load that causes Logstash to drop it`s UDP connections? So adding more than 2 machines will help?
Or is it something else entirely that can cause this thing?
Thanks,
Alex.