Logstash could not receive log after several minutes

Hi, my logstash is able to receive the log when the service is started. However after several minutes, it could not receive the log. I could not find any error message in syslog or logstash-plain.log.

I then run the /usr/share/logstash/bin/logstash --debug -f <config_folder> and get the error log as below (did not stop the service yet):

0001_input_windows_logstash.conf

input {
  tcp {
    port => 5001
    codec => json_lines
  }

9001_output_windows_logstash.conf

output {
  stdout {
    codec => rubydebug
  }
  rabbitmq {
    automatic_recovery => true
    exchange => "windows"
    exchange_type => "direct"
    host => <host>
    port => "5671"
    key => <key>
    user =><user>
    password => <password>
    durable => true
    persistent => true
    ssl => true
  }

error log

[ERROR] 2022-02-17 08:48:32.404 [[main]<tcp] javapipeline - A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::Tcp codec=><LogStash::Codecs::JSONLines id=>"json_lines_dc2f227c-160a-4370-af02-87042efccbf9", enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">, port=>5001, id=>"fc09b927c564b634fec8eb6e660087c1c021630a81ac5f445137b5b635867160", enable_metric=>true, host=>"0.0.0.0", mode=>"server", proxy_protocol=>false, ssl_enable=>false, ssl_verify=>true, ssl_key_passphrase=><password>, tcp_keep_alive=>false, dns_reverse_lookup_enabled=>true>
  Error: Address already in use
  Exception: Java::JavaNet::BindException
  Stack: sun.nio.ch.Net.bind0(Native Method)
sun.nio.ch.Net.bind(sun/nio/ch/Net.java:455)
sun.nio.ch.Net.bind(sun/nio/ch/Net.java:447)
sun.nio.ch.ServerSocketChannelImpl.bind(sun/nio/ch/ServerSocketChannelImpl.java:227)
io.netty.channel.socket.nio.NioServerSocketChannel.doBind(io/netty/channel/socket/nio/NioServerSocketChannel.java:134)
io.netty.channel.AbstractChannel$AbstractUnsafe.bind(io/netty/channel/AbstractChannel.java:562)
io.netty.channel.DefaultChannelPipeline$HeadContext.bind(io/netty/channel/DefaultChannelPipeline.java:1334)
io.netty.channel.AbstractChannelHandlerContext.invokeBind(io/netty/channel/AbstractChannelHandlerContext.java:506)
io.netty.channel.AbstractChannelHandlerContext.bind(io/netty/channel/AbstractChannelHandlerContext.java:491)
io.netty.channel.DefaultChannelPipeline.bind(io/netty/channel/DefaultChannelPipeline.java:973)
io.netty.channel.AbstractChannel.bind(io/netty/channel/AbstractChannel.java:260)
io.netty.bootstrap.AbstractBootstrap$2.run(io/netty/bootstrap/AbstractBootstrap.java:356)
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(io/netty/util/concurrent/AbstractEventExecutor.java:164)
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(io/netty/util/concurrent/SingleThreadEventExecutor.java:472)
io.netty.channel.nio.NioEventLoop.run(io/netty/channel/nio/NioEventLoop.java:500)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(io/netty/util/concurrent/SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(io/netty/util/internal/ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(io/netty/util/concurrent/FastThreadLocalRunnable.java:30)
java.lang.Thread.run(java/lang/Thread.java:829)
[DEBUG] 2022-02-17 08:48:33.408 [[main]<tcp] tcp - Closing {:plugin=>"LogStash::Inputs::Tcp"}
[DEBUG] 2022-02-17 08:48:34.478 [pool-5-thread-1] jvm - collector name {:name=>"ParNew"}
[DEBUG] 2022-02-17 08:48:34.483 [pool-5-thread-1] jvm - collector name {:name=>"ConcurrentMarkSweep"}
[DEBUG] 2022-02-17 08:48:36.423 [[main]<tcp] pluginmetadata - Removing metadata for plugin fc09b927c564b634fec8eb6e660087c1c021630a81ac5f445137b5b635867160
[INFO ] 2022-02-17 08:48:36.424 [[main]<tcp] tcp - Starting tcp input listener {:address=>"0.0.0.0:5001", :ssl_enable=>false}
[WARN ] 2022-02-17 08:48:36.429 [[main]<tcp] AbstractChannel - Force-closing a channel whose registration task was not accepted by an event loop: [id: 0xdb8cbeb6]
java.util.concurrent.RejectedExecutionException: event executor terminated
        at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926) ~[logstash-input-tcp-6.2.1.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353) ~[logstash-input-tcp-6.2.1.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346) ~[logstash-input-tcp-6.2.1.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828) ~[logstash-input-tcp-6.2.1.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818) ~[logstash-input-tcp-6.2.1.jar:?]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:483) [logstash-input-tcp-6.2.1.jar:?]
        at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:87) [logstash-input-tcp-6.2.1.jar:?]
        at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:81) [logstash-input-tcp-6.2.1.jar:?]
        at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86) [logstash-input-tcp-6.2.1.jar:?]
        at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:323) [logstash-input-tcp-6.2.1.jar:?]
        at io.netty.bootstrap.AbstractBootstrap.doBind(AbstractBootstrap.java:272) [logstash-input-tcp-6.2.1.jar:?]
        at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:268) [logstash-input-tcp-6.2.1.jar:?]
        at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:253) [logstash-input-tcp-6.2.1.jar:?]
        at org.logstash.tcp.InputLoop.run(InputLoop.java:86) [logstash-input-tcp-6.2.1.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:441) [jruby-complete-9.2.19.0.jar:?]
        at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:305) [jruby-complete-9.2.19.0.jar:?]
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:32) [jruby-complete-9.2.19.0.jar:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_tcp_minus_6_dot_2_dot_1_minus_java.lib.logstash.inputs.tcp.RUBY$method$run$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.2.1-java/lib/logstash/inputs/tcp.rb:160) [jruby-complete-9.2.19.0.jar:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_tcp_minus_6_dot_2_dot_1_minus_java.lib.logstash.inputs.tcp.RUBY$method$run$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.2.1-java/lib/logstash/inputs/tcp.rb) [jruby-complete-9.2.19.0.jar:?]
        at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80) [jruby-complete-9.2.19.0.jar:?]
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70) [jruby-complete-9.2.19.0.jar:?]
        at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207) [jruby-complete-9.2.19.0.jar:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$inputworker$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:405) [jruby-complete-9.2.19.0.jar:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$inputworker$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb) [jruby-complete-9.2.19.0.jar:?]
        at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80) [jruby-complete-9.2.19.0.jar:?]
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70) [jruby-complete-9.2.19.0.jar:?]
        at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207) [jruby-complete-9.2.19.0.jar:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_input$1(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:396) [jruby-complete-9.2.19.0.jar:?]
        at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138) [jruby-complete-9.2.19.0.jar:?]
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58) [jruby-complete-9.2.19.0.jar:?]
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52) [jruby-complete-9.2.19.0.jar:?]
        at org.jruby.runtime.Block.call(Block.java:139) [jruby-complete-9.2.19.0.jar:?]
        at org.jruby.RubyProc.call(RubyProc.java:318) [jruby-complete-9.2.19.0.jar:?]
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105) [jruby-complete-9.2.19.0.jar:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]

I have restarted the service with debug mode. I will paste the log once the logstash fails to receive the incoming log again.

Thanks.

After several minutes, the debug log will have such output as below:

It will stop the log ingestion and keep giving out the repeated output:

[DEBUG] 2022-02-17 09:32:05.223 [pool-6-thread-1] jvm - collector name {:name=>"ParNew"}
[DEBUG] 2022-02-17 09:32:05.223 [pool-6-thread-1] jvm - collector name {:name=>"ConcurrentMarkSweep"}
[DEBUG] 2022-02-17 09:32:10.227 [pool-5-thread-1] jvm - collector name {:name=>"ParNew"}
[DEBUG] 2022-02-17 09:32:10.227 [pool-5-thread-1] jvm - collector name {:name=>"ConcurrentMarkSweep"}
[DEBUG] 2022-02-17 09:32:15.232 [pool-6-thread-1] jvm - collector name {:name=>"ParNew"}
[DEBUG] 2022-02-17 09:32:15.232 [pool-6-thread-1] jvm - collector name {:name=>"ConcurrentMarkSweep"}
[DEBUG] 2022-02-17 09:32:20.236 [pool-5-thread-1] jvm - collector name {:name=>"ParNew"}
[DEBUG] 2022-02-17 09:32:20.236 [pool-5-thread-1] jvm - collector name {:name=>"ConcurrentMarkSweep"}
[DEBUG] 2022-02-17 09:32:25.240 [pool-6-thread-1] jvm - collector name {:name=>"ParNew"}
[DEBUG] 2022-02-17 09:32:25.241 [pool-6-thread-1] jvm - collector name {:name=>"ConcurrentMarkSweep"}
[DEBUG] 2022-02-17 09:32:30.244 [pool-5-thread-1] jvm - collector name {:name=>"ParNew"}
[DEBUG] 2022-02-17 09:32:30.245 [pool-5-thread-1] jvm - collector name {:name=>"ConcurrentMarkSweep"}
[DEBUG] 2022-02-17 09:32:35.248 [pool-6-thread-1] jvm - collector name {:name=>"ParNew"}
[DEBUG] 2022-02-17 09:32:35.249 [pool-6-thread-1] jvm - collector name {:name=>"ConcurrentMarkSweep"}
[DEBUG] 2022-02-17 09:32:40.253 [pool-5-thread-1] jvm - collector name {:name=>"ParNew"}
[DEBUG] 2022-02-17 09:32:40.253 [pool-5-thread-1] jvm - collector name {:name=>"ConcurrentMarkSweep"}
[DEBUG] 2022-02-17 09:32:45.258 [pool-6-thread-1] jvm - collector name {:name=>"ParNew"}
[DEBUG] 2022-02-17 09:32:45.258 [pool-6-thread-1] jvm - collector name {:name=>"ConcurrentMarkSweep"}
[DEBUG] 2022-02-17 09:32:50.262 [pool-5-thread-1] jvm - collector name {:name=>"ParNew"}
[DEBUG] 2022-02-17 09:32:50.262 [pool-5-thread-1] jvm - collector name {:name=>"ConcurrentMarkSweep"}
[DEBUG] 2022-02-17 09:32:55.266 [pool-6-thread-1] jvm - collector name {:name=>"ParNew"}
[DEBUG] 2022-02-17 09:32:55.266 [pool-6-thread-1] jvm - collector name {:name=>"ConcurrentMarkSweep"}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.