We're using an exec output to restart servers if tomcat has more than 40 stuck threads which we define as threads that take longer than 8 seconds. This recorded in a log file which logstash consumes. After tomcat restarts stuck threads reset to 0. Today we reached this limit and logstash did restart it; however, it restarted it several times in series.
The script that logstash calls blocks on the first call (while it waits to restart) and returns shortly on subsequent calls (by other threads) without actually restarting.
My best guess as to why this is happening is that the calls (or maybe the logs) are queued and cannot dequeue until the first call to the restart script finishes. Is this a reasonable guess? Currently I'm not backgrounding the exec script; would that fix this issue?
Here's a gist of the output:
Thanks,
Tyler
UPDATE:
Backgrounding with & does not work. I set up a test script and replaced the exec output script (/usr/local/bin/rq) with /usr/local/bin/test.sh.
[root@app101 conf.d]# cat /usr/local/bin/test.sh
#!/bin/bash
(date | tr '\n' '\t'; echo 'test') >> /tmp/test.out
sleep 5
And this is an example of the output of 9 matching log lines regardless of whether I background the process:
[root@app101 conf.d]# cat /tmp/test.out
Tue Jan 24 16:53:10 PST 2017 test
Tue Jan 24 16:53:15 PST 2017 test
Tue Jan 24 16:53:20 PST 2017 test
Tue Jan 24 16:53:25 PST 2017 test
Tue Jan 24 16:53:30 PST 2017 test
Tue Jan 24 16:53:35 PST 2017 test
Tue Jan 24 16:53:40 PST 2017 test
Tue Jan 24 16:53:45 PST 2017 test
Tue Jan 24 16:53:50 PST 2017 test
UPDATE:
I also tried setting the workers option of the exec output to 9 to no avail. Any other ideas?
UPDATE:
I am now using the throttle filter with success; however, it's not ideal since pipeline workers have been reduced to 1 from 8, and it may be causing logstash to crash.
{:timestamp=>"2017-01-25T11:06:05.386000-0800", :message=>"Defaulting pipeline worker threads to 1 because there are some filters that might not work with multiple worker threads", :count_was=>8, :filters=>["throttle"], :level=>:warn}
[root@app101 conf.d]# cat stuck_threads_filter.conf
filter {
if [type] == "TomcatThread" {
if [TotalStuckThreads] and [TotalStuckThreads] >= 40 {
throttle {
after_count => 1
period => 300
key => "%{type}"
add_tag => "throttled"
}
}
}
}
[root@app101 conf.d]# cat stuck_threads_output.conf
output {
if [type] == "TomcatThread" {
if [TotalStuckThreads] and [TotalStuckThreads] >= 40 and "throttled" not in [tags] {
exec {
command => "/usr/local/bin/test.sh"
}
}
}
}