Though filebeat is up and running we are unable to see the logs getting parsed under logstash. Deploying logstash config through centralized pipelines. logstash and filebeat are residing on different nodes.
Getting below error in logstash:
[2018-08-16T19:58:32,950][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:beats7
Plugin: <LogStash::Inputs::Beats port=>5044, id=>"3ecb088310f706f827e995c10532679a41540b96e3abd561f28616b5f5036c08", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_46e0da3e-dca4-4d60-822d-ee2cf8916338", enable_metric=>true, charset=>"UTF-8">, host=>"0.0.0.0", ssl=>false, ssl_verify_mode=>"none", include_codec_tag=>true, ssl_handshake_timeout=>10000, tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60, executor_threads=>6>
Error: event executor terminated
Exception: Java::JavaUtilConcurrent::RejectedExecutionException
Stack: io.netty.util.concurrent.SingleThreadEventExecutor.reject(io/netty/util/concurrent/SingleThreadEventExecutor.java:821)
io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(io/netty/util/concurrent/SingleThreadEventExecutor.java:327)
io.netty.util.concurrent.SingleThreadEventExecutor.addTask(io/netty/util/concurrent/SingleThreadEventExecutor.java:320)
io.netty.util.concurrent.SingleThreadEventExecutor.execute(io/netty/util/concurrent/SingleThreadEventExecutor.java:746)
io.netty.channel.AbstractChannel$AbstractUnsafe.register(io/netty/channel/AbstractChannel.java:479)
io.netty.channel.SingleThreadEventLoop.register(io/netty/channel/SingleThreadEventLoop.java:80)
io.netty.channel.SingleThreadEventLoop.register(io/netty/channel/SingleThreadEventLoop.java:74)
io.netty.channel.MultithreadEventLoopGroup.register(io/netty/channel/MultithreadEventLoopGroup.java:86)
io.netty.bootstrap.AbstractBootstrap.initAndRegister(io/netty/bootstrap/AbstractBootstrap.java:331)
io.netty.bootstrap.AbstractBootstrap.doBind(io/netty/bootstrap/AbstractBootstrap.java:282)
io.netty.bootstrap.AbstractBootstrap.bind(io/netty/bootstrap/AbstractBootstrap.java:278)
io.netty.bootstrap.AbstractBootstrap.bind(io/netty/bootstrap/AbstractBootstrap.java:260)
org.logstash.beats.Server.listen(org/logstash/beats/Server.java:57)
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:438)
org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:302)
opt.webmis.logstash_minus_6_dot_2_dot_4.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_beats_minus_5_dot_0_dot_13_minus_java.lib.logstash.inputs.beats.run(/usr/elk/logstash-6.2.4/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.0.13-java/lib/logstash/inputs/beats.rb:198)
opt.webmis.logstash_minus_6_dot_2_dot_4.logstash_minus_core.lib.logstash.pipeline.inputworker(/usr/elk/logstash-6.2.4/logstash-core/lib/logstash/pipeline.rb:514)
opt.webmis.logstash_minus_6_dot_2_dot_4.logstash_minus_core.lib.logstash.pipeline.RUBY$method$inputworker$0$VARARGS(/usr/elk/logstash_minus_6_dot_2_dot_4/logstash_minus_core/lib/logstash//usr/elk/logstash-6.2.4/logstash-core/lib/logstash/pipeline.rb)
opt.webmis.logstash_minus_6_dot_2_dot_4.logstash_minus_core.lib.logstash.pipeline.block in start_input(/usr/elk/logstash-6.2.4/logstash-core/lib/logstash/pipeline.rb:507)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:246)
java.lang.Thread.run(java/lang/Thread.java:748)
[2018-10-16T19:58:33,950][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2018-10-16T19:58:33,951][WARN ][io.netty.channel.AbstractChannel] Force-closing a channel whose registration task was not accepted by an event loop: [id: 0x67834446]
java.util.concurrent.RejectedExecutionException: event executor terminated