Logstash fails to start listener due to Error: failed to create a child event loop in io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(io/netty/util/concurrent/MultithreadEventExecutorGroup.java:88)

tlukac · August 21, 2023, 7:26pm

Logstash continually raises this error when attempting to start LogStash::Inputs::Beats plugin. There is no "Caused by" info in the stack trace so cannot determine what is causing the issue. Any ideas on how to diagnose this further would be appreciated. I've tried several things like completely disabling firewall, changing host/port values to no avail:

  Pipeline_id:main
  Plugin: <LogStash::Inputs::Beats ssl_certificate=>"/root/logstash/liberty/logstash.crt", id=>"aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861", ssl_key=>"/root/logstash/liberty/logstash.key", port=>5043, ssl=>true, enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_e1fef20a-bc88-4c75-b500-5a250e2f6462", enable_metric=>true, charset=>"UTF-8">, host=>"0.0.0.0", add_hostname=>false, ssl_verify_mode=>"none", ssl_peer_metadata=>false, include_codec_tag=>true, ssl_handshake_timeout=>10000, tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60, executor_threads=>8>
  Error: failed to create a child event loop
  Exception: Java::JavaLang::IllegalStateException
  Stack: io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(io/netty/util/concurrent/MultithreadEventExecutorGroup.java:88)
io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(io/netty/util/concurrent/MultithreadEventExecutorGroup.java:60)
io.netty.channel.MultithreadEventLoopGroup.<init>(io/netty/channel/MultithreadEventLoopGroup.java:52)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:96)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:91)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:72)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:52)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:44)
org.logstash.beats.Server.listen(org/logstash/beats/Server.java:48)
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)
org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:441)
org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:305)
root.logstash.logstash_minus_7_dot_17_dot_12.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_2_dot_6_minus_java.lib.logstash.inputs.beats.run(/root/logstash/logstash-7.17.12/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.2.6-java/lib/logstash/inputs/beats.rb:201)
root.logstash.logstash_minus_7_dot_17_dot_12.logstash_minus_core.lib.logstash.java_pipeline.inputworker(/root/logstash/logstash-7.17.12/logstash-core/lib/logstash/java_pipeline.rb:410)
root.logstash.logstash_minus_7_dot_17_dot_12.logstash_minus_core.lib.logstash.java_pipeline.start_input(/root/logstash/logstash-7.17.12/logstash-core/lib/logstash/java_pipeline.rb:401)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:318)
java.lang.Thread.run(java/lang/Thread.java:829)
[2023-08-21T12:41:05,181][DEBUG][logstash.inputs.beats    ][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] Closing {:plugin=>"LogStash::Inputs::Beats"}
[2023-08-21T12:41:05,181][DEBUG][logstash.pluginmetadata  ][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] Removing metadata for plugin aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861
[2023-08-21T12:41:07,198][ERROR][logstash.javapipeline    ][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] A plugin had an unrecoverable error. Will restart this plugin.

leandrojmp · August 21, 2023, 9:38pm

Hello and welcome,

Please share the beginning of the log, you shared just part of it, the log line starts with a timestamp followed by the log level.

Also, share your logstash configuration for your input.

tlukac · August 22, 2023, 1:09am

Here is the configuration file:

input {
    beats {
        port => "5043"
	ssl => "true"
        ssl_certificate => "/root/logstash/liberty/logstash.crt" 
        ssl_key => "/root/logstash/liberty/logstash.key" 
    }
}
filter {
    if [type] =~ "liberty_*" {
        if [type] == "liberty_accesslog" {
            json {
                source => "message"
                remove_field => [ "message" ]
            }
        }
        else if [type] == "liberty_trace" {
            json {
                source => "message"
            }
        }
        else if [type] == "liberty_gc" {
            json {
                source => "message"
                remove_field => [ "message" ]
            }
        }
        else if [type] == "liberty_message" {
            json {
                source => "message"
            }
        }
        else if [type] == "liberty_ffdc" {
            json {
                source => "message"
                remove_field => [ "message" ]
            }
        }
        else if [type] == "liberty_audit" {
            json {
                source => "message"
                remove_field => [ "message" ]
            }
        }
    }
}
output {
    elasticsearch {
        hosts => "b4smd2-dom0.dr.avaya.com:9200"
        template => "/root/logstash/liberty/liberty_logstash_template_ibm.json"
        template_overwrite => "true"
    }
}

I also tried with setting host in input to 127.0.0.1 but didn't make any difference.

Here's more of the log output as you requested:

[2023-08-21T12:40:58,937][DEBUG][org.logstash.netty.SslContextBuilder][main] Available ciphers: [TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
[2023-08-21T12:40:58,937][DEBUG][org.logstash.netty.SslContextBuilder][main] Ciphers:  [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256]
[2023-08-21T12:40:58,942][DEBUG][io.netty.handler.ssl.OpenSsl][main] netty-tcnative not in the classpath; OpenSslEngine will be unavailable.
[2023-08-21T12:40:58,965][DEBUG][io.netty.handler.ssl.JdkSslContext][main] Default protocols (JDK): [TLSv1.3, TLSv1.2] 
[2023-08-21T12:40:58,965][DEBUG][io.netty.handler.ssl.JdkSslContext][main] Default cipher suites (JDK): [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384]
[2023-08-21T12:40:59,051][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2023-08-21T12:40:59,057][DEBUG][logstash.javapipeline    ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x5f92b366 run>"}
[2023-08-21T12:40:59,064][DEBUG][org.logstash.execution.PeriodicFlush][main] Pushing flush onto pipeline.
[2023-08-21T12:40:59,089][DEBUG][io.netty.channel.MultithreadEventLoopGroup][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] -Dio.netty.eventLoopThreads: 16
[2023-08-21T12:40:59,104][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2023-08-21T12:40:59,115][DEBUG][io.netty.channel.nio.NioEventLoop][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] -Dio.netty.noKeySetOptimization: false
[2023-08-21T12:40:59,115][DEBUG][io.netty.channel.nio.NioEventLoop][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] -Dio.netty.selectorAutoRebuildThreshold: 512
[2023-08-21T12:40:59,123][DEBUG][io.netty.util.internal.PlatformDependent][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] org.jctools-core.MpscChunkedArrayQueue: available
[2023-08-21T12:41:01,155][ERROR][logstash.javapipeline    ][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::Beats ssl_certificate=>"/root/logstash/liberty/logstash.crt", id=>"aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861", ssl_key=>"/root/logstash/liberty/logstash.key", port=>5043, ssl=>true, enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_e1fef20a-bc88-4c75-b500-5a250e2f6462", enable_metric=>true, charset=>"UTF-8">, host=>"0.0.0.0", add_hostname=>false, ssl_verify_mode=>"none", ssl_peer_metadata=>false, include_codec_tag=>true, ssl_handshake_timeout=>10000, tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60, executor_threads=>8>
  Error: failed to create a child event loop
  Exception: Java::JavaLang::IllegalStateException
  Stack: io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(io/netty/util/concurrent/MultithreadEventExecutorGroup.java:88)
io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(io/netty/util/concurrent/MultithreadEventExecutorGroup.java:60)
io.netty.channel.MultithreadEventLoopGroup.<init>(io/netty/channel/MultithreadEventLoopGroup.java:52)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:96)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:91)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:72)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:52)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:44)
org.logstash.beats.Server.listen(org/logstash/beats/Server.java:48)
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)
org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:426)
org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:293)
root.logstash.logstash_minus_7_dot_17_dot_12.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_2_dot_6_minus_java.lib.logstash.inputs.beats.run(/root/logstash/logstash-7.17.12/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.2.6-java/lib/logstash/inputs/beats.rb:201)
root.logstash.logstash_minus_7_dot_17_dot_12.logstash_minus_core.lib.logstash.java_pipeline.inputworker(/root/logstash/logstash-7.17.12/logstash-core/lib/logstash/java_pipeline.rb:410)
root.logstash.logstash_minus_7_dot_17_dot_12.logstash_minus_core.lib.logstash.java_pipeline.start_input(/root/logstash/logstash-7.17.12/logstash-core/lib/logstash/java_pipeline.rb:401)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:318)
java.lang.Thread.run(java/lang/Thread.java:829)
[2023-08-21T12:41:02,162][DEBUG][logstash.inputs.beats    ][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] Closing {:plugin=>"LogStash::Inputs::Beats"}
[2023-08-21T12:41:02,165][DEBUG][logstash.pluginmetadata  ][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] Removing metadata for plugin aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861
[2023-08-21T12:41:03,269][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2023-08-21T12:41:03,270][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2023-08-21T12:41:04,063][DEBUG][org.logstash.execution.PeriodicFlush][main] Pushing flush onto pipeline.
[2023-08-21T12:41:04,179][ERROR][logstash.javapipeline    ][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::Beats ssl_certificate=>"/root/logstash/liberty/logstash.crt", id=>"aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861", ssl_key=>"/root/logstash/liberty/logstash.key", port=>5043, ssl=>true, enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_e1fef20a-bc88-4c75-b500-5a250e2f6462", enable_metric=>true, charset=>"UTF-8">, host=>"0.0.0.0", add_hostname=>false, ssl_verify_mode=>"none", ssl_peer_metadata=>false, include_codec_tag=>true, ssl_handshake_timeout=>10000, tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60, executor_threads=>8>
  Error: failed to create a child event loop
  Exception: Java::JavaLang::IllegalStateException
  Stack: io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(io/netty/util/concurrent/MultithreadEventExecutorGroup.java:88)
io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(io/netty/util/concurrent/MultithreadEventExecutorGroup.java:60)
io.netty.channel.MultithreadEventLoopGroup.<init>(io/netty/channel/MultithreadEventLoopGroup.java:52)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:96)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:91)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:72)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:52)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:44)
org.logstash.beats.Server.listen(org/logstash/beats/Server.java:48)
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)
org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:441)
org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:305)
root.logstash.logstash_minus_7_dot_17_dot_12.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_2_dot_6_minus_java.lib.logstash.inputs.beats.run(/root/logstash/logstash-7.17.12/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.2.6-java/lib/logstash/inputs/beats.rb:201)
root.logstash.logstash_minus_7_dot_17_dot_12.logstash_minus_core.lib.logstash.java_pipeline.inputworker(/root/logstash/logstash-7.17.12/logstash-core/lib/logstash/java_pipeline.rb:410)
root.logstash.logstash_minus_7_dot_17_dot_12.logstash_minus_core.lib.logstash.java_pipeline.start_input(/root/logstash/logstash-7.17.12/logstash-core/lib/logstash/java_pipeline.rb:401)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:318)
java.lang.Thread.run(java/lang/Thread.java:829)
[2023-08-21T12:41:05,181][DEBUG][logstash.inputs.beats    ][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] Closing {:plugin=>"LogStash::Inputs::Beats"}
[2023-08-21T12:41:05,181][DEBUG][logstash.pluginmetadata  ][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] Removing metadata for plugin aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861

tlukac · August 29, 2023, 8:24pm

I get the error only when starting logstash via systemctl. When running from command line the error doesn't occur.

leandrojmp · August 29, 2023, 8:51pm

This may be the issue.

When you run logstash using systemctl, it will be executed using the logstash user, which will not have permission to read anything in the /root path, so it cannot read the certificates used in the input configuration.

When running from command line are you running as the root user? You should not run logstash as the root user.

You need to do the following:

Move the .crt. and .key files to a path where the logstash user can read it and change the paths in your input configuration.
Start logstash with systemctl and check if you will get any new error, it is probably that some directories have the wrong permissions if you run logstash as root before, check the permissions for /var/lib/logstash and /var/log/logstash.

tlukac · August 30, 2023, 4:50pm

Actually I ran it as root and as different (non-root) user and that didn't make any difference. After some additional looking around I found that the service definition in /etc/systemd/system/logstash.service contained the line:

LimitNOFILE=128

in the [Service] section. I increased this value to 4096 and things started to work. No longer getting the IllegalStateException. Not sure how that LimitNoFILE value got there as I didn't add it -- maybe through the logstash system-install?

Badger · August 30, 2023, 6:23pm

I am pretty sure my /usr/lib/systemd/system/logstash.service is from the logstash install and it has LimitNOFILE=16384.

Rios · September 1, 2023, 9:54am

Just to confirm, a fresh LS installation( 8.9.1): LimitNOFILE=16384

system · September 29, 2023, 9:54am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
ERROR logstash/async.go:256 Failed to publish events caused by: lumberjack protocol error & client is not connected Logstash	1	1114	December 4, 2019
Unable to start logstash Logstash	3	1618	November 5, 2020
Logstash unable to listen to Beats Error: event executor terminated Logstash	3	2620	October 4, 2017
Logstash TCP and Syslog Plugin Error Logstash	5	1123	June 6, 2023
Logstash-output-syslog Logstash	1	501	May 1, 2018

Logstash fails to start listener due to Error: failed to create a child event loop in io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(io/netty/util/concurrent/MultithreadEventExecutorGroup.java:88)

Related topics