ShutdownWatcherExt The shutdown process appears to be stalled due to busy or blocked plugins without input entrace

Hello, I'm installing new stack elk for my stage env.
Kibana, and ES are ok, but I have problem with logstash.
Currently, I have no input on this env. no metricbeat started, no filebeat, nothing.
I configured pipelines, to prepare all my input, start logstash without problem, but when I stop it:

> [2020-06-30T16:47:55,934][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>35, "name"=>"[main]<beats", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.9-java/lib/logstash/inputs/beats.rb:197:in `run'"}, {"thread_id"=>36, "name"=>"[main]<pipeline", "current_call"=>"[...]/logstash-core/lib/logstash/plugins/builtin/pipeline/input.rb:43:in `sleep'"}, {"thread_id"=>37, "name"=>"[main]<tcp", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.0.4-java/lib/logstash/inputs/tcp.rb:153:in `run'"}, {"thread_id"=>38, "name"=>"[main]<pipeline", "current_call"=>"[...]/logstash-core/lib/logstash/plugins/builtin/pipeline/input.rb:43:in `sleep'"}, {"thread_id"=>39, "name"=>"[main]<pipeline", "current_call"=>"[...]/logstash-core/lib/logstash/plugins/builtin/pipeline/input.rb:43:in `sleep'"}], ["LogStash::Filters::Mutate", {"add_field"=>{"[@metadata][index_name]"=>"metricbeat_%{+YYYY-MM-dd}"}, "id"=>"7a465fd96a234ea6bd32b267ed00c1c9ce9d1c1544a74aaaa53b34c42e202ab7"}]=>[{"thread_id"=>31, "name"=>"[main]>worker0", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:279:in `block in start_workers'"}, {"thread_id"=>32, "name"=>"[main]>worker1", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:279:in `block in start_workers'"}, {"thread_id"=>33, "name"=>"[main]>worker2", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:279:in `block in start_workers'"}, {"thread_id"=>34, "name"=>"[main]>worker3", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:279:in `block in start_workers'"}]}}
> [2020-06-30T16:47:55,946][ERROR][org.logstash.execution.ShutdownWatcherExt] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.
> [2020-06-30T16:48:00,995][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>36, "name"=>"[main]<pipeline", "current_call"=>"[...]/logstash-core/lib/logstash/plugins/builtin/pipeline/input.rb:43:in `sleep'"}, {"thread_id"=>37, "name"=>"[main]<tcp", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.0.4-java/lib/logstash/inputs/tcp.rb:153:in `run'"}, {"thread_id"=>38, "name"=>"[main]<pipeline", "current_call"=>"[...]/logstash-core/lib/logstash/plugins/builtin/pipeline/input.rb:43:in `sleep'"}, {"thread_id"=>39, "name"=>"[main]<pipeline", "current_call"=>"[...]/logstash-core/lib/logstash/plugins/builtin/pipeline/input.rb:43:in `sleep'"}], ["LogStash::Filters::Mutate", {"add_field"=>{"[@metadata][index_name]"=>"metricbeat_%{+YYYY-MM-dd}"}, "id"=>"7a465fd96a234ea6bd32b267ed00c1c9ce9d1c1544a74aaaa53b34c42e202ab7"}]=>[{"thread_id"=>31, "name"=>"[main]>worker0", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:279:in `block in start_workers'"}, {"thread_id"=>32, "name"=>"[main]>worker1", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:279:in `block in start_workers'"}, {"thread_id"=>33, "name"=>"[main]>worker2", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:279:in `block in start_workers'"}, {"thread_id"=>34, "name"=>"[main]>worker3", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:279:in `block in start_workers'"}]}}

I have to kill the logstash for it stop.
Why logstashis busy and eat nothing ?

here is the metricbeat pipeline

> # metric-beat.pipeline.conf
> input { pipeline { address => metricbeat } }
> filter {
>    mutate { add_field => { "[@metadata][index_name]" => "metricbeat_%{+YYYY-MM-dd}" } }
> }
> output {
>   pipeline { send_to => sortie }
> }

Do you have an idea ?
Thanks

1 Like

my pipelines are simple:
1 dispatcher
N to transform data
1 output

The problem was order of pipelines.
I keeped the pipeline.yml to *.conf.
I had to use one line per pipeline file. with correct order (input > N transform data > output)

2 Likes

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.