Hi Team,
I am trying to use tcp plugin in logstash. When i run the config file related to this using logstash cli command , i am able to start it without any issues. However when i try starting it as a systemd service i am getting below error. There are no port conflicts , i've verified that.
This issue is somewhat similar to https://discuss.elastic.co/t/logstash-http-input-having-unrecoverable-error-when-logstash-is-used-in-systemd/199161/5 .
While running as systemd service , logstash starts but not tcp port
[ash@ logstash] netstat -tnpl | egrep "9600|514"
(No info could be read for "-p": geteuid()=5842 but you should be root.)
tcp6 0 0 :::9600 :::* LISTEN -
[ash@logstash]
Error
[2021-03-12T03:47:15,698][ERROR][logstash.javapipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:network-logs
Plugin: <LogStash::Inputs::Tcp type=>"syslog", port=>514, id=>"1a47f26295ff147be386caec8bb103df7d4141d6fc554c06d6505431672abb97", enable_metric=>true, codec=><LogStash::Codecs::Line id=>"line_e8d6b675-53de-4d2c-9dc1-c14fcae3d0d5", enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">, host=>"0.0.0.0", mode=>"server", proxy_protocol=>false, ssl_enable=>false, ssl_verify=>true, ssl_key_passphrase=>, tcp_keep_alive=>false, dns_reverse_lookup_enabled=>true>
Error: event executor terminated
Exception: Java::JavaUtilConcurrent::RejectedExecutionException
Stack: io.netty.util.concurrent.SingleThreadEventExecutor.reject(io/netty/util/concurrent/SingleThreadEventExecutor.java:821)
io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(io/netty/util/concurrent/SingleThreadEventExecutor.java:327)
io.netty.util.concurrent.SingleThreadEventExecutor.addTask(io/netty/util/concurrent/SingleThreadEventExecutor.java:320)
io.netty.util.concurrent.SingleThreadEventExecutor.execute(io/netty/util/concurrent/SingleThreadEventExecutor.java:746)
io.netty.channel.AbstractChannel$AbstractUnsafe.register(io/netty/channel/AbstractChannel.java:479)
io.netty.channel.SingleThreadEventLoop.register(io/netty/channel/SingleThreadEventLoop.java:80)
[2021-03-12T03:41:38,595][WARN ][io.netty.channel.AbstractChannel] Force-closing a channel whose registration task was not accepted by an event loop: [id: 0xd45cde14]
java.util.concurrent.RejectedExecutionException: event executor terminated
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:821) ~[logstash-input-tcp-6.0.5.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:327) ~[logstash-input-tcp-6.0.5.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:320) ~[logstash-input-tcp-6.0.5.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:746) ~[logstash-input-tcp-6.0.5.jar:?]
Config File:
more /etc/logstash/conf.d/network-logs.conf
input {
tcp {
port => 514
type => syslog
}
}
But if i run it from CLI using logstash command, there are no issues. Plz help
sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/network-logs.conf
Thread.exclusive is deprecated, use Thread::Mutex
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2021-03-12 03:52:09.760 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2021-03-12 03:52:09.773 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.3.0"}
[INFO ] 2021-03-12 03:52:14.105 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2021-03-12 03:52:14.130 [[main]<tcp] tcp - Starting tcp input listener {:address=>"0.0.0.0:514", :ssl_enable=>"false"}
[INFO ] 2021-03-12 03:52:14.229 [Ruby-0-Thread-2: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[INFO ] 2021-03-12 03:52:14.548 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}