Hi, my logstash is able to receive the log when the service is started. However after several minutes, it could not receive the log. I could not find any error message in syslog or logstash-plain.log.
I then run the /usr/share/logstash/bin/logstash --debug -f <config_folder> and get the error log as below (did not stop the service yet):
0001_input_windows_logstash.conf
input {
tcp {
port => 5001
codec => json_lines
}
9001_output_windows_logstash.conf
output {
stdout {
codec => rubydebug
}
rabbitmq {
automatic_recovery => true
exchange => "windows"
exchange_type => "direct"
host => <host>
port => "5671"
key => <key>
user =><user>
password => <password>
durable => true
persistent => true
ssl => true
}
error log
[ERROR] 2022-02-17 08:48:32.404 [[main]<tcp] javapipeline - A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::Tcp codec=><LogStash::Codecs::JSONLines id=>"json_lines_dc2f227c-160a-4370-af02-87042efccbf9", enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">, port=>5001, id=>"fc09b927c564b634fec8eb6e660087c1c021630a81ac5f445137b5b635867160", enable_metric=>true, host=>"0.0.0.0", mode=>"server", proxy_protocol=>false, ssl_enable=>false, ssl_verify=>true, ssl_key_passphrase=><password>, tcp_keep_alive=>false, dns_reverse_lookup_enabled=>true>
Error: Address already in use
Exception: Java::JavaNet::BindException
Stack: sun.nio.ch.Net.bind0(Native Method)
sun.nio.ch.Net.bind(sun/nio/ch/Net.java:455)
sun.nio.ch.Net.bind(sun/nio/ch/Net.java:447)
sun.nio.ch.ServerSocketChannelImpl.bind(sun/nio/ch/ServerSocketChannelImpl.java:227)
io.netty.channel.socket.nio.NioServerSocketChannel.doBind(io/netty/channel/socket/nio/NioServerSocketChannel.java:134)
io.netty.channel.AbstractChannel$AbstractUnsafe.bind(io/netty/channel/AbstractChannel.java:562)
io.netty.channel.DefaultChannelPipeline$HeadContext.bind(io/netty/channel/DefaultChannelPipeline.java:1334)
io.netty.channel.AbstractChannelHandlerContext.invokeBind(io/netty/channel/AbstractChannelHandlerContext.java:506)
io.netty.channel.AbstractChannelHandlerContext.bind(io/netty/channel/AbstractChannelHandlerContext.java:491)
io.netty.channel.DefaultChannelPipeline.bind(io/netty/channel/DefaultChannelPipeline.java:973)
io.netty.channel.AbstractChannel.bind(io/netty/channel/AbstractChannel.java:260)
io.netty.bootstrap.AbstractBootstrap$2.run(io/netty/bootstrap/AbstractBootstrap.java:356)
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(io/netty/util/concurrent/AbstractEventExecutor.java:164)
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(io/netty/util/concurrent/SingleThreadEventExecutor.java:472)
io.netty.channel.nio.NioEventLoop.run(io/netty/channel/nio/NioEventLoop.java:500)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(io/netty/util/concurrent/SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(io/netty/util/internal/ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(io/netty/util/concurrent/FastThreadLocalRunnable.java:30)
java.lang.Thread.run(java/lang/Thread.java:829)
[DEBUG] 2022-02-17 08:48:33.408 [[main]<tcp] tcp - Closing {:plugin=>"LogStash::Inputs::Tcp"}
[DEBUG] 2022-02-17 08:48:34.478 [pool-5-thread-1] jvm - collector name {:name=>"ParNew"}
[DEBUG] 2022-02-17 08:48:34.483 [pool-5-thread-1] jvm - collector name {:name=>"ConcurrentMarkSweep"}
[DEBUG] 2022-02-17 08:48:36.423 [[main]<tcp] pluginmetadata - Removing metadata for plugin fc09b927c564b634fec8eb6e660087c1c021630a81ac5f445137b5b635867160
[INFO ] 2022-02-17 08:48:36.424 [[main]<tcp] tcp - Starting tcp input listener {:address=>"0.0.0.0:5001", :ssl_enable=>false}
[WARN ] 2022-02-17 08:48:36.429 [[main]<tcp] AbstractChannel - Force-closing a channel whose registration task was not accepted by an event loop: [id: 0xdb8cbeb6]
java.util.concurrent.RejectedExecutionException: event executor terminated
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926) ~[logstash-input-tcp-6.2.1.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353) ~[logstash-input-tcp-6.2.1.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346) ~[logstash-input-tcp-6.2.1.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828) ~[logstash-input-tcp-6.2.1.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818) ~[logstash-input-tcp-6.2.1.jar:?]
at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:483) [logstash-input-tcp-6.2.1.jar:?]
at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:87) [logstash-input-tcp-6.2.1.jar:?]
at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:81) [logstash-input-tcp-6.2.1.jar:?]
at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86) [logstash-input-tcp-6.2.1.jar:?]
at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:323) [logstash-input-tcp-6.2.1.jar:?]
at io.netty.bootstrap.AbstractBootstrap.doBind(AbstractBootstrap.java:272) [logstash-input-tcp-6.2.1.jar:?]
at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:268) [logstash-input-tcp-6.2.1.jar:?]
at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:253) [logstash-input-tcp-6.2.1.jar:?]
at org.logstash.tcp.InputLoop.run(InputLoop.java:86) [logstash-input-tcp-6.2.1.jar:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:441) [jruby-complete-9.2.19.0.jar:?]
at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:305) [jruby-complete-9.2.19.0.jar:?]
at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:32) [jruby-complete-9.2.19.0.jar:?]
at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_tcp_minus_6_dot_2_dot_1_minus_java.lib.logstash.inputs.tcp.RUBY$method$run$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.2.1-java/lib/logstash/inputs/tcp.rb:160) [jruby-complete-9.2.19.0.jar:?]
at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_tcp_minus_6_dot_2_dot_1_minus_java.lib.logstash.inputs.tcp.RUBY$method$run$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.2.1-java/lib/logstash/inputs/tcp.rb) [jruby-complete-9.2.19.0.jar:?]
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80) [jruby-complete-9.2.19.0.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70) [jruby-complete-9.2.19.0.jar:?]
at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207) [jruby-complete-9.2.19.0.jar:?]
at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$inputworker$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:405) [jruby-complete-9.2.19.0.jar:?]
at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$inputworker$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb) [jruby-complete-9.2.19.0.jar:?]
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80) [jruby-complete-9.2.19.0.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70) [jruby-complete-9.2.19.0.jar:?]
at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207) [jruby-complete-9.2.19.0.jar:?]
at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_input$1(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:396) [jruby-complete-9.2.19.0.jar:?]
at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138) [jruby-complete-9.2.19.0.jar:?]
at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58) [jruby-complete-9.2.19.0.jar:?]
at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52) [jruby-complete-9.2.19.0.jar:?]
at org.jruby.runtime.Block.call(Block.java:139) [jruby-complete-9.2.19.0.jar:?]
at org.jruby.RubyProc.call(RubyProc.java:318) [jruby-complete-9.2.19.0.jar:?]
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105) [jruby-complete-9.2.19.0.jar:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
I have restarted the service with debug mode. I will paste the log once the logstash fails to receive the incoming log again.
Thanks.