SIGKILL twice to kill or use SIGINT and SIGTERM

Elastic Stack Logstash
#1

It seems logstash (technically launched via java) traps the SIGKILL unix signal (ie. kill -9) and tries to shutdown gracefully. However, I have experienced times when it never shuts down.

I suggest that the policy be that if while handing the first SIGKILL another is received, the app simply exit with data loss. Another policy would be to property use SIGINT and/or SIGTERM instead of SIGKILL for graceful exit and SIGKILL for what it is designed for - killing the process. Otherwise, when the process hangs, I can only get rid of it by rebooting the machine.

FYI: I am running pipe plugin as a logstash input with the tail command:
input {
pipe {
add_field => { "source" => "foo.log" }
command => "tail -f ./logs/foo.log"
}
And killing the main process leaves 4 zombies attached to the root pid, so ps -ef | grep java still shows the logstash java process even after several kill -9's.

Using logstash 2.3.1 on Solaris 10

#2

Good morning,
in logstash specifically we don't trap the SIGKILL, you can see how do we react to the different signals at https://github.com/elastic/logstash/blob/master/logstash-core/lib/logstash/runner.rb#L368-L394

On the other side, we don't really test in solaris, so can not help much on the behaviour there, might this be related to the way the pipe input you're using works in solaris?

see

skywalker% ps -ax | grep logstash
 1830 ttys000    0:27.88 /Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home//bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/Users/purbon/work/logstash/vendor/jruby/lib/jni -Dfile.encoding=UTF-8 -Xbootclasspath/a:/Users/purbon/work/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/Users/purbon/work/logstash/vendor/jruby -Djruby.lib=/Users/purbon/work/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /Users/purbon/work/logstash/lib/bootstrap/environment.rb logstash/runner.rb -e 
 1957 ttys002    0:00.00 grep logstash
skywalker% kill -9 1830
skywalker% ps -ax | grep logstash
 1959 ttys002    0:00.00 grep logstash
skywalker%
#3