Logstash Stops Very Frequently

Elastic Stack Logstash
1

Using the version 5.6.2 of Logstash, and I see lot of times the following message in the logstash-json.log file

{"level":"ERROR","loggerName":"logstash.shutdownwatcher","timeMillis":1723464009728,"thread":"Ruby-0-Thread-504: /usr/share/logstash/logstash-core/lib/logstash/shutdown_watcher.rb:31","logEvent":{"message":"The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information."}}

Please help

2

Hello and welcome,

Version 5.6.2 is too old, there is no support for it anymore and a lot has changed since it was released.

The supported versions are 7.17.23 and 8.15.0, please see if you can upgrade.

Are you using persistent queues? The error you shared normally happens when logstash received a shutdown command, but still has to drain the date in the persistent queues.

3

If you are getting that message from ShutdownWatcherExt you should also be getting a message that points you towards the issue. For example....

[org.logstash.execution.ShutdownWatcherExt] {
"inflight_count"=>1, "stalling_threads_info"=>{[...], "id"=>"..."}]=>[{"thread_id"=>68, "name"=>"[main]>worker0",
"current_call"=>"[...]/vendor/bundle/jruby/3.1.0/gems/logstash-output-tcp-6.2.1/lib/logstash/outputs/tcp.rb:355:in `sleep'"}], ...

Here it is a tcp output stuck in a loop trying to connect to a remote server (a tcp output will not shut down until it has successfully started up). It will loop forever printing the same error messages again and again.