Hello,
I'm experiencing a Logstash error with the syslog input plugin. The input plugin for my pipeline keeps crashing with the message Force-closing a channel whose registration task was not accepted by an event loop and A plugin had an unrecoverable error. Will restart this plugin.
This results in only the udp port being opened but not both tcp and udp (which should be the normal behaviour if I'm not mistaken).
There is no other pipeline that is configured to use this port, so there shouldn't be any conflicts.
This is the error I get from logstash:
[2023-04-25T10:58:19,696][INFO ][logstash.inputs.tcp ][my_log_source][d443af5bc08bf4f9af84081172b5891f490cb8e57db0a7e2575e719f4143fbc6] Starting tcp input listener {:address=>"192.168.0.35:49166", :ssl_enable=>false}
[2023-04-25T10:58:19,697][WARN ][io.netty.channel.AbstractChannel][my_log_source][d443af5bc08bf4f9af84081172b5891f490cb8e57db0a7e2575e719f4143fbc6] Force-closing a channel whose registration task was not accepted by an event loop: [id: 0x7b1445e1]
java.util.concurrent.RejectedExecutionException: event executor terminated
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:934) ~[netty-common-4.1.87.Final.jar:4.1.87.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:351) ~[netty-common-4.1.87.Final.jar:4.1.87.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:344) ~[netty-common-4.1.87.Final.jar:4.1.87.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:836) ~[netty-common-4.1.87.Final.jar:4.1.87.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:827) ~[netty-common-4.1.87.Final.jar:4.1.87.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:817) ~[netty-common-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:483) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:89) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:83) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:323) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.bootstrap.AbstractBootstrap.doBind(AbstractBootstrap.java:272) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:268) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:253) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at org.logstash.tcp.InputLoop.run(InputLoop.java:86) [logstash-input-tcp-6.3.2.jar:?]
at jdk.internal.reflect.GeneratedMethodAccessor153.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:442) [jruby.jar:?]
at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:306) [jruby.jar:?]
at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:32) [jruby.jar:?]
at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_input_minus_tcp_minus_6_dot_3_dot_2_minus_java.lib.logstash.inputs.tcp.RUBY$method$run$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-tcp-6.3.2-java/lib/logstash/inputs/tcp.rb:167) [?:?]
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:165) [jruby.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:185) [jruby.jar:?]
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:218) [jruby.jar:?]
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:173) [jruby.jar:?]
at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316) [jruby.jar:?]
at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72) [jruby.jar:?]
at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:86) [jruby.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:201) [jruby.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:188) [jruby.jar:?]
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:218) [jruby.jar:?]
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:372) [jruby.jar:?]
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:175) [jruby.jar:?]
at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316) [jruby.jar:?]
at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72) [jruby.jar:?]
at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116) [jruby.jar:?]
at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136) [jruby.jar:?]
at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66) [jruby.jar:?]
at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58) [jruby.jar:?]
at org.jruby.runtime.Block.call(Block.java:143) [jruby.jar:?]
at org.jruby.RubyProc.call(RubyProc.java:309) [jruby.jar:?]
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:107) [jruby.jar:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-04-25T10:58:19,698][ERROR][logstash.javapipeline ][my_log_source][d443af5bc08bf4f9af84081172b5891f490cb8e57db0a7e2575e719f4143fbc6] A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:my_log_source
Plugin: <LogStash::Inputs::Tcp host=>"192.168.0.35", id=>"d443af5bc08bf4f9af84081172b5891f490cb8e57db0a7e2575e719f4143fbc6", port=>49166, tags=>["my_log_source"], enable_metric=>true, codec=><LogStash::Codecs::Line id=>"line_ec431109-d7f2-4cb3-91d9-ae831163ee41", enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">, mode=>"server", proxy_protocol=>false, ssl_enable=>false, ssl_verify=>true, ssl_key_passphrase=><password>, tcp_keep_alive=>false, dns_reverse_lookup_enabled=>true>
Error: event executor terminated
Exception: Java::JavaUtilConcurrent::RejectedExecutionException
Stack: io.netty.util.concurrent.SingleThreadEventExecutor.reject(io/netty/util/concurrent/SingleThreadEventExecutor.java:934)
io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(io/netty/util/concurrent/SingleThreadEventExecutor.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor.addTask(io/netty/util/concurrent/SingleThreadEventExecutor.java:344)
io.netty.util.concurrent.SingleThreadEventExecutor.execute(io/netty/util/concurrent/SingleThreadEventExecutor.java:836)
io.netty.util.concurrent.SingleThreadEventExecutor.execute0(io/netty/util/concurrent/SingleThreadEventExecutor.java:827)
io.netty.util.concurrent.SingleThreadEventExecutor.execute(io/netty/util/concurrent/SingleThreadEventExecutor.java:817)
io.netty.channel.AbstractChannel$AbstractUnsafe.register(io/netty/channel/AbstractChannel.java:483)
io.netty.channel.SingleThreadEventLoop.register(io/netty/channel/SingleThreadEventLoop.java:89)
io.netty.channel.SingleThreadEventLoop.register(io/netty/channel/SingleThreadEventLoop.java:83)
io.netty.channel.MultithreadEventLoopGroup.register(io/netty/channel/MultithreadEventLoopGroup.java:86)
io.netty.bootstrap.AbstractBootstrap.initAndRegister(io/netty/bootstrap/AbstractBootstrap.java:323)
io.netty.bootstrap.AbstractBootstrap.doBind(io/netty/bootstrap/AbstractBootstrap.java:272)
io.netty.bootstrap.AbstractBootstrap.bind(io/netty/bootstrap/AbstractBootstrap.java:268)
io.netty.bootstrap.AbstractBootstrap.bind(io/netty/bootstrap/AbstractBootstrap.java:253)
org.logstash.tcp.InputLoop.run(org/logstash/tcp/InputLoop.java:86)
jdk.internal.reflect.GeneratedMethodAccessor153.invoke(jdk/internal/reflect/GeneratedMethodAccessor153)
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:568)
org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:442)
org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:306)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_input_minus_tcp_minus_6_dot_3_dot_2_minus_java.lib.logstash.inputs.tcp.run(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-tcp-6.3.2-java/lib/logstash/inputs/tcp.rb:167)
RUBY.inputworker(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:414)
RUBY.start_input(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:405)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:309)
java.lang.Thread.run(java/lang/Thread.java:833)
The problem also occurs with other pipelines, as if they were taking turns, and there is always at least one pipeline, that throws these errors. Sometimes it also happens with the tcp input plugin, but never with udp. I've already tried restarting the server (Ubuntu 22.04 lts) but the issue persists.