Failed to submit a listener notification task... A plugin had an unrecoverable error

Elastic Stack Logstash
1

I am seeing this error in logstash every few seconds. any help how to fix this? I am running 5.0-alpha5.

20:34:34.228 [[main]<beats] ERROR io.netty.util.concurrent.DefaultPromise.rejectedExecution - Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:840) ~[netty-all-4.1.3.Final.jar:4.1.3.Final]

..
at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:436) [?:?]
at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:300) [?:?]
at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:35) [?:?]
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134) [?:?]
at rubyjit.LogStash::Inputs::Beats$$run_d71aefd1f73bd440f7e6d0b3603d8ed9fb9fcbcd1173230247.file(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-3.1.0.beta4-java/lib/logstash/inputs/beats.rb:169) [?:?]
at rubyjit.LogStash::Inputs::Beats$$run_d71aefd1f73bd440f7e6d0b3603d8ed9fb9fcbcd1173230247.file(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-3.1.0.beta4-java/lib/logstash/inputs/beats.rb) [?:?]
at org.jruby.internal.runtime.methods.JittedMethod.call(JittedMethod.java:181) [?:?]
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168) [?:?]
...
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99) [?:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]
{:timestamp=>"2016-09-19T20:34:34.230000+0000", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::Beats port=>5044, enable_metric=>true, codec=><LogStash::Codecs::Plain enable_metric=>true, charset=>"UTF-8">, host=>"0.0.0.0", ssl=>false, ssl_verify_mode=>"none", ssl_handshake_timeout=>10000, congestion_threshold=>5, target_field_for_codec=>"message", tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA38", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>15>\n Error: event executor terminated", :level=>:error}
{:timestamp=>"2016-09-19T20:34:34.505000+0000", :message=>"SNMP Trap listener died", :exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in bind'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/snmp-1.2.0/lib/snmp/manager.rb:540:ininitialize'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/snmp-1.2.0/lib/snmp/manager.rb:585:in create_transport'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/snmp-1.2.0/lib/snmp/manager.rb:618:ininitialize'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-snmptrap-3.0.2/lib/logstash/inputs/snmptrap.rb:74:in build_trap_listener'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-snmptrap-3.0.2/lib/logstash/inputs/snmptrap.rb:78:insnmptrap_listener'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-snmptrap-3.0.2/lib/logstash/inputs/snmptrap.rb:53:in run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:323:ininputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:317:in `start_input'"], :level=>:warn}

2

What's your config look like?

3 
input {
        beats {
                port => 5044
        }

        snmptrap {
                type => "snmptrap"
                community => "public"
                port => 8162
        }
}

### Add your filters / logstash plugins configuration here

filter {
  if "_grokparsefailure" in [tags] {
        drop {}
  }
}

output {
        elasticsearch {
                hosts => "elasticsearch:9200"
                index => "%{[index]}-%{+YYYY.MM.dd}"
                document_type => "%{type}"
        }
}
4

I'm having exactly the same error when starting logstash with a Beats input. Logstash works ok with a file input. Any progress here ?

I'm using logstash 5.0.0-rc1.

5

Was there ever a solution for this? Running into this now.

6

Same error here, running filebeat-5.1.1-darwin-x86_64 and logstash-5.1.1. Any suggestion?

Follow the instruction: https://www.elastic.co/guide/en/beats/filebeat/current/config-filebeat-logstash.html

7

Hi, I'm seeing this error too

filebeat-5.1.1-darwin-x86_64 and logstash-5.1.1.

8

Yeah, I'm having this issue as well. Would love to get some feedback on this.

9

PS to all, didn't get this working for 5.1, but I downgraded to 5.0.1 and it seems to work fine again.