# Logstash stalling during auto-reconfigure even though inflight\_events = 0 **URL:** \ **Category:** Logstash\ **Created:** [March 24, 2018, 3:55am UTC](https://discuss.elastic.co/t/logstash-stalling-during-auto-reconfigure-even-though-inflight-events-0/125423 "2018-03-24T03:55:55Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Crisps](https://avatars.discourse-cdn.com/v4/letter/c/ee7513/32.png) [@Crisps](https://discuss.elastic.co/u/Crisps)\ **Post date:** [March 24, 2018, 3:55am UTC](https://discuss.elastic.co/t/logstash-stalling-during-auto-reconfigure-even-though-inflight-events-0/125423/1 "2018-03-24T03:55:55Z")
Hello, I have logstash so that it will autoreconfigure. When it reconfigures, it works fine sometimes, however, sometimes it will randomly give the message `[ERROR][logstash.shutdownwatcher] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.` And then I can see that the restart is stalling with 0 inflight\_count: `[2018-03-24T03:54:15,150][WARN][logstash.shutdownwatcher] {"inflight_count"=>0, "stalling_thread_info"=>{["LogStash::Filters::Mutate", {"rename"=>{"a"=>"%{b}"}, "id"=>"f0bc5038d58971608846f9c9deca78509541b842fcd91bde2f37e2aa531b8acd"}]=>[{"thread_id"=>54, "name"=>nil, "current_call"=>"[...]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:133:in`initialize'"}, {"thread\_id"=\>55, "name"=\>nil, "current\_call"=\>"[...]/logstash-core/lib/logstash/util/wrapped\_synchronous\_queue.rb:133:in `initialize'"}]}} [2018-03-24T03:54:20,149][WARN][logstash.shutdownwatcher] {"inflight_count"=>0, "stalling_thread_info"=>{["LogStash::Filters::Mutate", {"rename"=>{"a"=>"%{b}"}, "id"=>"f0bc5038d58971608846f9c9deca78509541b842fcd91bde2f37e2aa531b8acd"}]=>[{"thread_id"=>54, "name"=>nil, "current_call"=>"[...]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:133:in`initialize'"}, {"thread\_id"=\>55, "name"=\>nil, "current\_call"=\>"[...]/logstash-core/lib/logstash/util/wrapped\_synchronous\_queue.rb:133:in `initialize'"}]}} [2018-03-24T03:54:25,150][WARN][logstash.shutdownwatcher] {"inflight_count"=>0, "stalling_thread_info"=>{["LogStash::Filters::Mutate", {"rename"=>{"a"=>"%{b}"}, "id"=>"f0bc5038d58971608846f9c9deca78509541b842fcd91bde2f37e2aa531b8acd"}]=>[{"thread_id"=>54, "name"=>nil, "current_call"=>"[...]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:133:in`initialize'"}, {"thread\_id"=\>55, "name"=\>nil, "current\_call"=\>"[...]/logstash-core/lib/logstash/util/wrapped\_synchronous\_queue.rb:133:in `initialize'"}]}} [2018-03-24T03:54:30,150][WARN][logstash.shutdownwatcher] {"inflight_count"=>0, "stalling_thread_info"=>{["LogStash::Filters::Mutate", {"rename"=>{"a"=>"%{b}"}, "id"=>"f0bc5038d58971608846f9c9deca78509541b842fcd91bde2f37e2aa531b8acd"}]=>[{"thread_id"=>54, "name"=>nil, "current_call"=>"[...]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:133:in`initialize'"}, {"thread\_id"=\>55, "name"=\>nil, "current\_call"=\>"[...]/logstash-core/lib/logstash/util/wrapped\_synchronous\_queue.rb:133:in `initialize'"}]}}` It looks like it is the last line in my filter that it is stalling at and it seems to do this regardless of what the content of the last line actually is. I'd appreciate some help, thanks! ---
**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\ **Post date:** [March 24, 2018, 5:42am UTC](https://discuss.elastic.co/t/logstash-stalling-during-auto-reconfigure-even-though-inflight-events-0/125423/2 "2018-03-24T05:42:43Z")
We recently found a race condition in how pipelines are shutdown, and have a fix in a [pull request](https://github.com/elastic/logstash/pull/9285) as of earlier today. It should make it into 6.3.0 in the next couple weeks. ---
**Author:** ![Crisps](https://avatars.discourse-cdn.com/v4/letter/c/ee7513/32.png) [@Crisps](https://discuss.elastic.co/u/Crisps)\ **Post date:** [March 24, 2018, 7:54am UTC](https://discuss.elastic.co/t/logstash-stalling-during-auto-reconfigure-even-though-inflight-events-0/125423/3 "2018-03-24T07:54:19Z")
Thanks for the info. I'm running Logstash with those changes in right now and it seems to no longer stall, but, I get another error occasionally during reconfiguration: > [2018-03-24T07:50:01,774][FATAL][logstash.runner] An unexpected error occurred! {:error=\>#\, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/util.rb:40:in `thread\_info'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:668:in `block in plugin_threads_info'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:668:in `plugin_threads_info'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:672:in `stalling\_threads\_info'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline\_reporter.rb:67:in `block in to_hash'", "/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:80:in `inflight\_batches'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline\_reporter.rb:56:in `to_hash'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_reporter.rb:51:in `snapshot'", "/usr/share/logstash/logstash-core/lib/logstash/shutdown\_watcher.rb:88:in `pipeline_report_snapshot'", "/usr/share/logstash/logstash-core/lib/logstash/shutdown_watcher.rb:63:in `block in start'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:20:in `interval'", "/usr/share/logstash/logstash-core/lib/logstash/shutdown_watcher.rb:59:in `start'", "/usr/share/logstash/logstash-core/lib/logstash/shutdown\_watcher.rb:35:in `block in start'"]} Looks like it's caused by `backtrace = thread.backtrace.map do |line|` in thread\_info of util.rb? Is there a nil check missing for thread.backtrace? ---
**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\ **Post date:** [March 24, 2018, 8:10am UTC](https://discuss.elastic.co/t/logstash-stalling-during-auto-reconfigure-even-though-inflight-events-0/125423/4 "2018-03-24T08:10:41Z")
d'oh! I found that one too and had a PR sitting unmerged until just now: > ---
**Author:** ![Crisps](https://avatars.discourse-cdn.com/v4/letter/c/ee7513/32.png) [@Crisps](https://discuss.elastic.co/u/Crisps)\ **Post date:** [March 24, 2018, 8:15am UTC](https://discuss.elastic.co/t/logstash-stalling-during-auto-reconfigure-even-though-inflight-events-0/125423/5 "2018-03-24T08:15:49Z")
Looks like it's been a productive day for the Elastic team! Thanks a lot! ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 10, 2018, 5:18pm UTC](https://discuss.elastic.co/t/logstash-stalling-during-auto-reconfigure-even-though-inflight-events-0/125423/6 "2018-04-10T17:18:36Z")
Is pipeline.workers=1 a workaround for this? ---
**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\ **Post date:** [April 10, 2018, 10:06pm UTC](https://discuss.elastic.co/t/logstash-stalling-during-auto-reconfigure-even-though-inflight-events-0/125423/7 "2018-04-10T22:06:21Z")
> [@Badger](#): > > Is pipeline.workers=1 a workaround for this? _probably?_ I've not done the mental gymnastics to validate, but on the surface I think this would likely eliminate the race condition. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 8, 2018, 10:06pm UTC](https://discuss.elastic.co/t/logstash-stalling-during-auto-reconfigure-even-though-inflight-events-0/125423/8 "2018-05-08T22:06:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.