Process in stall state while shutdown

Elastic Stack Logstash
1

We have a script to stop the logstash, which inturn runs 'kill -15 '. This script is being called as part of system init file which in turn stops as part of server reboot.

However, recently we see issues in prod while stopping logstash process. Where system stop is failing with timeout exception. We tried increasing the timeout but still getting the same exception.

Tested the same in dev host and logstash is not stopping instead we are getting the below warning [even after two hours of running kill command].

Kindly let us know, any reason and solution for the same.

I also saw similar topic raised but dont see any proper solution for the same. In one of them i see, the user fixed jruby but not sure what exactly was done. [The shutdown process appears to be stalled due to busy or blocked plugins]

P.S : Logstash version is 5.6.10

Regards, Vinutha

2

Look in the logs to see if there are additional messages. If not, get a thread dump from the JVM and see which threads are hanging. If you are unable to tell that from the dump post it as a gist or on some other file sharing site and I will take a look.

3

I do see 'messages in flight due to mutate filter' in log entry along with the 'stall due to busy or blocked plugins' exceptions.

We haven't changed anything at the logstash end and it was working fine from years. Recently, we started seeing this issue when we stop logstash i.e kill -15

4

Not a filter I would expect to hang. Can you post the thread dump?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.