Good day!
I have an issue with my Logstash.
I have many JMS inputs and one Elasticsearch output.
Output config is simple:
elasticsearch {
hosts => "${LS_ES_HOSTS}"
ssl => true
manage_template => false
document_type => "%{[@metadata][type]}"
index => "%{[@metadata][indexName]}-%{+YYYY.MM.dd}"
}
Logstash outputs data with proxy server and sometimes, when proxy dies Logstash not able to send data to Elasticsearch and it crashes with error (busy/blocked plugin):
[2019-05-14T13:49:38,143][WARN ][org.logstash.execution.ShutdownWatcherExt]
{"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>109,
"name"=>"[main]<jms", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/jruby-jms-1.3.0-java/lib/jms/connection.rb:308:in `block in close'"}],
["LogStash::Filters::Mutate", {"lowercase"=>["[@metadata][indexName]"], "id"=>"f68936d423b088a2d10d8b3accc8a9ef7ee8fd8b149a6b4d8466bba807d9a409"}]
=>[{"thread_id"=>26, "name"=>"[main]>worker0", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"},
{"thread_id"=>27, "name"=>"[main]>worker1", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}]}}
[2019-05-14T13:49:38,148][ERROR][org.logstash.execution.ShutdownWatcherExt] The shutdown process appears to be stalled due to busy or blocked plugins.
The worst thing is even proxy server coming up, Logstash will have "infinity reboot" status.
Are there any solutions to "ping" Elasticsearch server and if it is unreachable - stop Logstash processing?