Logstash fails to start listener due to Error: failed to create a child event loop in io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(io/netty/util/concurrent/MultithreadEventExecutorGroup.java:88)

Logstash continually raises this error when attempting to start LogStash::Inputs::Beats plugin. There is no "Caused by" info in the stack trace so cannot determine what is causing the issue. Any ideas on how to diagnose this further would be appreciated. I've tried several things like completely disabling firewall, changing host/port values to no avail:

  Pipeline_id:main
  Plugin: <LogStash::Inputs::Beats ssl_certificate=>"/root/logstash/liberty/logstash.crt", id=>"aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861", ssl_key=>"/root/logstash/liberty/logstash.key", port=>5043, ssl=>true, enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_e1fef20a-bc88-4c75-b500-5a250e2f6462", enable_metric=>true, charset=>"UTF-8">, host=>"0.0.0.0", add_hostname=>false, ssl_verify_mode=>"none", ssl_peer_metadata=>false, include_codec_tag=>true, ssl_handshake_timeout=>10000, tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60, executor_threads=>8>
  Error: failed to create a child event loop
  Exception: Java::JavaLang::IllegalStateException
  Stack: io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(io/netty/util/concurrent/MultithreadEventExecutorGroup.java:88)
io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(io/netty/util/concurrent/MultithreadEventExecutorGroup.java:60)
io.netty.channel.MultithreadEventLoopGroup.<init>(io/netty/channel/MultithreadEventLoopGroup.java:52)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:96)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:91)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:72)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:52)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:44)
org.logstash.beats.Server.listen(org/logstash/beats/Server.java:48)
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)
org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:441)
org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:305)
root.logstash.logstash_minus_7_dot_17_dot_12.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_2_dot_6_minus_java.lib.logstash.inputs.beats.run(/root/logstash/logstash-7.17.12/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.2.6-java/lib/logstash/inputs/beats.rb:201)
root.logstash.logstash_minus_7_dot_17_dot_12.logstash_minus_core.lib.logstash.java_pipeline.inputworker(/root/logstash/logstash-7.17.12/logstash-core/lib/logstash/java_pipeline.rb:410)
root.logstash.logstash_minus_7_dot_17_dot_12.logstash_minus_core.lib.logstash.java_pipeline.start_input(/root/logstash/logstash-7.17.12/logstash-core/lib/logstash/java_pipeline.rb:401)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:318)
java.lang.Thread.run(java/lang/Thread.java:829)
[2023-08-21T12:41:05,181][DEBUG][logstash.inputs.beats    ][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] Closing {:plugin=>"LogStash::Inputs::Beats"}
[2023-08-21T12:41:05,181][DEBUG][logstash.pluginmetadata  ][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] Removing metadata for plugin aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861
[2023-08-21T12:41:07,198][ERROR][logstash.javapipeline    ][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] A plugin had an unrecoverable error. Will restart this plugin.

Hello and welcome,

Please share the beginning of the log, you shared just part of it, the log line starts with a timestamp followed by the log level.

Also, share your logstash configuration for your input.

Here is the configuration file:

input {
    beats {
        port => "5043"
	ssl => "true"
        ssl_certificate => "/root/logstash/liberty/logstash.crt" 
        ssl_key => "/root/logstash/liberty/logstash.key" 
    }
}
filter {
    if [type] =~ "liberty_*" {
        if [type] == "liberty_accesslog" {
            json {
                source => "message"
                remove_field => [ "message" ]
            }
        }
        else if [type] == "liberty_trace" {
            json {
                source => "message"
            }
        }
        else if [type] == "liberty_gc" {
            json {
                source => "message"
                remove_field => [ "message" ]
            }
        }
        else if [type] == "liberty_message" {
            json {
                source => "message"
            }
        }
        else if [type] == "liberty_ffdc" {
            json {
                source => "message"
                remove_field => [ "message" ]
            }
        }
        else if [type] == "liberty_audit" {
            json {
                source => "message"
                remove_field => [ "message" ]
            }
        }
    }
}
output {
    elasticsearch {
        hosts => "b4smd2-dom0.dr.avaya.com:9200"
        template => "/root/logstash/liberty/liberty_logstash_template_ibm.json"
        template_overwrite => "true"
    }
}

I also tried with setting host in input to 127.0.0.1 but didn't make any difference.

Here's more of the log output as you requested:

[2023-08-21T12:40:58,937][DEBUG][org.logstash.netty.SslContextBuilder][main] Available ciphers: [TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
[2023-08-21T12:40:58,937][DEBUG][org.logstash.netty.SslContextBuilder][main] Ciphers:  [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256]
[2023-08-21T12:40:58,942][DEBUG][io.netty.handler.ssl.OpenSsl][main] netty-tcnative not in the classpath; OpenSslEngine will be unavailable.
[2023-08-21T12:40:58,965][DEBUG][io.netty.handler.ssl.JdkSslContext][main] Default protocols (JDK): [TLSv1.3, TLSv1.2] 
[2023-08-21T12:40:58,965][DEBUG][io.netty.handler.ssl.JdkSslContext][main] Default cipher suites (JDK): [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384]
[2023-08-21T12:40:59,051][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2023-08-21T12:40:59,057][DEBUG][logstash.javapipeline    ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x5f92b366 run>"}
[2023-08-21T12:40:59,064][DEBUG][org.logstash.execution.PeriodicFlush][main] Pushing flush onto pipeline.
[2023-08-21T12:40:59,089][DEBUG][io.netty.channel.MultithreadEventLoopGroup][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] -Dio.netty.eventLoopThreads: 16
[2023-08-21T12:40:59,104][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2023-08-21T12:40:59,115][DEBUG][io.netty.channel.nio.NioEventLoop][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] -Dio.netty.noKeySetOptimization: false
[2023-08-21T12:40:59,115][DEBUG][io.netty.channel.nio.NioEventLoop][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] -Dio.netty.selectorAutoRebuildThreshold: 512
[2023-08-21T12:40:59,123][DEBUG][io.netty.util.internal.PlatformDependent][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] org.jctools-core.MpscChunkedArrayQueue: available
[2023-08-21T12:41:01,155][ERROR][logstash.javapipeline    ][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::Beats ssl_certificate=>"/root/logstash/liberty/logstash.crt", id=>"aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861", ssl_key=>"/root/logstash/liberty/logstash.key", port=>5043, ssl=>true, enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_e1fef20a-bc88-4c75-b500-5a250e2f6462", enable_metric=>true, charset=>"UTF-8">, host=>"0.0.0.0", add_hostname=>false, ssl_verify_mode=>"none", ssl_peer_metadata=>false, include_codec_tag=>true, ssl_handshake_timeout=>10000, tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60, executor_threads=>8>
  Error: failed to create a child event loop
  Exception: Java::JavaLang::IllegalStateException
  Stack: io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(io/netty/util/concurrent/MultithreadEventExecutorGroup.java:88)
io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(io/netty/util/concurrent/MultithreadEventExecutorGroup.java:60)
io.netty.channel.MultithreadEventLoopGroup.<init>(io/netty/channel/MultithreadEventLoopGroup.java:52)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:96)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:91)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:72)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:52)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:44)
org.logstash.beats.Server.listen(org/logstash/beats/Server.java:48)
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)
org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:426)
org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:293)
root.logstash.logstash_minus_7_dot_17_dot_12.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_2_dot_6_minus_java.lib.logstash.inputs.beats.run(/root/logstash/logstash-7.17.12/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.2.6-java/lib/logstash/inputs/beats.rb:201)
root.logstash.logstash_minus_7_dot_17_dot_12.logstash_minus_core.lib.logstash.java_pipeline.inputworker(/root/logstash/logstash-7.17.12/logstash-core/lib/logstash/java_pipeline.rb:410)
root.logstash.logstash_minus_7_dot_17_dot_12.logstash_minus_core.lib.logstash.java_pipeline.start_input(/root/logstash/logstash-7.17.12/logstash-core/lib/logstash/java_pipeline.rb:401)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:318)
java.lang.Thread.run(java/lang/Thread.java:829)
[2023-08-21T12:41:02,162][DEBUG][logstash.inputs.beats    ][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] Closing {:plugin=>"LogStash::Inputs::Beats"}
[2023-08-21T12:41:02,165][DEBUG][logstash.pluginmetadata  ][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] Removing metadata for plugin aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861
[2023-08-21T12:41:03,269][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2023-08-21T12:41:03,270][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2023-08-21T12:41:04,063][DEBUG][org.logstash.execution.PeriodicFlush][main] Pushing flush onto pipeline.
[2023-08-21T12:41:04,179][ERROR][logstash.javapipeline    ][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::Beats ssl_certificate=>"/root/logstash/liberty/logstash.crt", id=>"aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861", ssl_key=>"/root/logstash/liberty/logstash.key", port=>5043, ssl=>true, enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_e1fef20a-bc88-4c75-b500-5a250e2f6462", enable_metric=>true, charset=>"UTF-8">, host=>"0.0.0.0", add_hostname=>false, ssl_verify_mode=>"none", ssl_peer_metadata=>false, include_codec_tag=>true, ssl_handshake_timeout=>10000, tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60, executor_threads=>8>
  Error: failed to create a child event loop
  Exception: Java::JavaLang::IllegalStateException
  Stack: io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(io/netty/util/concurrent/MultithreadEventExecutorGroup.java:88)
io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(io/netty/util/concurrent/MultithreadEventExecutorGroup.java:60)
io.netty.channel.MultithreadEventLoopGroup.<init>(io/netty/channel/MultithreadEventLoopGroup.java:52)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:96)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:91)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:72)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:52)
io.netty.channel.nio.NioEventLoopGroup.<init>(io/netty/channel/nio/NioEventLoopGroup.java:44)
org.logstash.beats.Server.listen(org/logstash/beats/Server.java:48)
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)
org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:441)
org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:305)
root.logstash.logstash_minus_7_dot_17_dot_12.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_2_dot_6_minus_java.lib.logstash.inputs.beats.run(/root/logstash/logstash-7.17.12/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.2.6-java/lib/logstash/inputs/beats.rb:201)
root.logstash.logstash_minus_7_dot_17_dot_12.logstash_minus_core.lib.logstash.java_pipeline.inputworker(/root/logstash/logstash-7.17.12/logstash-core/lib/logstash/java_pipeline.rb:410)
root.logstash.logstash_minus_7_dot_17_dot_12.logstash_minus_core.lib.logstash.java_pipeline.start_input(/root/logstash/logstash-7.17.12/logstash-core/lib/logstash/java_pipeline.rb:401)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:318)
java.lang.Thread.run(java/lang/Thread.java:829)
[2023-08-21T12:41:05,181][DEBUG][logstash.inputs.beats    ][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] Closing {:plugin=>"LogStash::Inputs::Beats"}
[2023-08-21T12:41:05,181][DEBUG][logstash.pluginmetadata  ][main][aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861] Removing metadata for plugin aae8b2eee90d3b297a0e5872fc87c0a8073274f67e68c2c7cfecfefb61957861

I get the error only when starting logstash via systemctl. When running from command line the error doesn't occur.

This may be the issue.

When you run logstash using systemctl, it will be executed using the logstash user, which will not have permission to read anything in the /root path, so it cannot read the certificates used in the input configuration.

When running from command line are you running as the root user? You should not run logstash as the root user.

You need to do the following:

  • Move the .crt. and .key files to a path where the logstash user can read it and change the paths in your input configuration.
  • Start logstash with systemctl and check if you will get any new error, it is probably that some directories have the wrong permissions if you run logstash as root before, check the permissions for /var/lib/logstash and /var/log/logstash.

Actually I ran it as root and as different (non-root) user and that didn't make any difference. After some additional looking around I found that the service definition in /etc/systemd/system/logstash.service contained the line:

LimitNOFILE=128

in the [Service] section. I increased this value to 4096 and things started to work. No longer getting the IllegalStateException. Not sure how that LimitNoFILE value got there as I didn't add it -- maybe through the logstash system-install?

I am pretty sure my /usr/lib/systemd/system/logstash.service is from the logstash install and it has LimitNOFILE=16384.

Just to confirm, a fresh LS installation( 8.9.1): LimitNOFILE=16384

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.