# Nonstop Lumberjack pipeline is blocked messages

**URL:** <https://discuss.elastic.co/t/nonstop-lumberjack-pipeline-is-blocked-messages/25246>\
**Category:** Logstash\
**Created:** [July 9, 2015, 3:13pm UTC](https://discuss.elastic.co/t/nonstop-lumberjack-pipeline-is-blocked-messages/25246 "2015-07-09T15:13:15Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![JeremyinNC](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeremyinnc/32/44868_2.png) [@JeremyinNC](https://discuss.elastic.co/u/JeremyinNC)\
**Post date:** [July 9, 2015, 3:13pm UTC](https://discuss.elastic.co/t/nonstop-lumberjack-pipeline-is-blocked-messages/25246/1 "2015-07-09T15:13:15Z")

</div>

I recently upgraded Logstash from 1.4 to 1.5.1 and now I am constantly seeing the following in my logstash log:

```
{:timestamp=>"2015-07-09T15:10:03.559000+0000", :message=>"Lumberjack input: the pipeline is blocked, temporary refusing new connection.", :level=>:warn}
{:timestamp=>"2015-07-09T15:10:03.761000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Lumberjack input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn}
{:timestamp=>"2015-07-09T15:10:03.761000+0000", :message=>"CircuitBreaker::Open", :name=>"Lumberjack input", :level=>:warn}
{:timestamp=>"2015-07-09T15:10:03.761000+0000", :message=>"Exception in lumberjack input thread", :exception=>#<LogStash::CircuitBreaker::OpenBreaker: for Lumberjack input>, :level=>:error}
{:timestamp=>"2015-07-09T15:10:04.059000+0000", :message=>"Lumberjack input: the pipeline is blocked, temporary refusing new connection.", :level=>:warn}

```

Does this just mean I am receiving data faster than I can write it out to Elasticsearch? The service is staying up but I am not sure what this message means.

---

<div class="post-metadata">

**Author:** ![NightWolf](https://avatars.discourse-cdn.com/v4/letter/n/779978/32.png) [@NightWolf](https://discuss.elastic.co/u/NightWolf)\
**Post date:** [July 13, 2015, 3:13am UTC](https://discuss.elastic.co/t/nonstop-lumberjack-pipeline-is-blocked-messages/25246/2 "2015-07-13T03:13:25Z")

</div>

+1 seeing this same issue. Did you find what was causing it?

My log is just full of this spam. Running LS 1.5.2

```
{:timestamp=>"2015-07-13T01:14:15.154000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Lumberjack input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn}
{:timestamp=>"2015-07-13T01:14:15.155000+0000", :message=>"CircuitBreaker::Open", :name=>"Lumberjack input", :level=>:warn}
{:timestamp=>"2015-07-13T01:14:15.157000+0000", :message=>"Exception in lumberjack input thread", :exception=>#<LogStash::CircuitBreaker::OpenBreaker: for Lumberjack input>, :level=>:error}
{:timestamp=>"2015-07-13T01:14:15.454000+0000", :message=>"CircuitBreaker::Open", :name=>"Lumberjack input", :level=>:warn}
{:timestamp=>"2015-07-13T01:14:15.455000+0000", :message=>"Exception in lumberjack input thread", :exception=>#<LogStash::CircuitBreaker::OpenBreaker: for Lumberjack input>, :level=>:error}
{:timestamp=>"2015-07-13T01:15:02.262000+0000", :message=>"CircuitBreaker::Close", :name=>"Lumberjack input", :level=>:warn}
```

---

<div class="post-metadata">

**Author:** ![MartinT](https://avatars.discourse-cdn.com/v4/letter/m/ee7513/32.png) [@MartinT](https://discuss.elastic.co/u/MartinT)\
**Post date:** [July 13, 2015, 3:00pm UTC](https://discuss.elastic.co/t/nonstop-lumberjack-pipeline-is-blocked-messages/25246/3 "2015-07-13T15:00:07Z")

</div>

Hello, i had a exactly the same problem. In my case the issue was a misconfiguration in logstash \<\> ES. ES was configured to listen on interface IP, but logstash try to connect them on localhost. Immediately after fix this typo and restart ES and Logstash everything is OK.  
Hopefully, this will help you also as a hint

---

<div class="post-metadata">

**Author:** ![JeremyinNC](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeremyinnc/32/44868_2.png) [@JeremyinNC](https://discuss.elastic.co/u/JeremyinNC)\
**Post date:** [July 21, 2015, 3:14pm UTC](https://discuss.elastic.co/t/nonstop-lumberjack-pipeline-is-blocked-messages/25246/4 "2015-07-21T15:14:52Z")

</div>

Did any events make it into Elasticsearch? Events (lots of them!) are being pushed into ES. I think this is a message you get when ever Logstash is trying to throttle events from the clients.

---

<div class="post-metadata">

**Author:** ![panaman](https://avatars.discourse-cdn.com/v4/letter/p/3da27b/32.png) [@panaman](https://discuss.elastic.co/u/panaman)\
**Post date:** [July 22, 2015, 12:47pm UTC](https://discuss.elastic.co/t/nonstop-lumberjack-pipeline-is-blocked-messages/25246/5 "2015-07-22T12:47:46Z")

</div>

i had this happen to me last night (logstash 1.5.2)  
All events stopped going into elasticsearch. Logstash looked like it was running, but it wasn't doing anything.  
I had nothing in my logstash.err, but had a bunch of these in my logstash.log servers.  
{:timestamp=\>"2015-07-22T00:02:53.274000+0000", :message=\>"Lumberjack input: the pipeline is blocked, temporary refusing new connection.", :level=\>:warn}

I ended up having to restart logstash to get it all flowing again.

---

<div class="post-metadata">

**Author:** ![iamrudra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iamrudra/32/4160_2.png) [@iamrudra](https://discuss.elastic.co/u/iamrudra)\
**Post date:** [August 12, 2015, 4:34pm UTC](https://discuss.elastic.co/t/nonstop-lumberjack-pipeline-is-blocked-messages/25246/6 "2015-08-12T16:34:45Z")

</div>

For me the events didn't go into the elasticsearch. This happened when I put another `elasticsearch` block in the `output` plugin. I commented it out and the behavior was back to normal.

---

<div class="post-metadata">

**Author:** ![panaman](https://avatars.discourse-cdn.com/v4/letter/p/3da27b/32.png) [@panaman](https://discuss.elastic.co/u/panaman)\
**Post date:** [August 24, 2015, 1:32am UTC](https://discuss.elastic.co/t/nonstop-lumberjack-pipeline-is-blocked-messages/25246/7 "2015-08-24T01:32:26Z")

</div>

this constantly happens to me, events stop going into elasticsearch when this happens. All i can do is restart logstash to make it work again. Verified that this is happening on 1.5.4.  
I have a 24 cpu thread logstash server and 16 workers. Also to note when this happens, the box is idle not using any resources.

---

<div class="post-metadata">

**Author:** ![D\_Luffy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d_luffy/32/4687_2.png) [@D\_Luffy](https://discuss.elastic.co/u/D_Luffy)\
**Post date:** [September 10, 2015, 4:14am UTC](https://discuss.elastic.co/t/nonstop-lumberjack-pipeline-is-blocked-messages/25246/8 "2015-09-10T04:14:03Z")

</div>

I am experiencing the exact same behaviour, crawled all the forums out there but couldn't find a fix.

do we have a fix at all? thanks!

happy to make code changes.

---

<div class="post-metadata">

**Author:** ![JeremyinNC](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeremyinnc/32/44868_2.png) [@JeremyinNC](https://discuss.elastic.co/u/JeremyinNC)\
**Post date:** [September 10, 2015, 4:21pm UTC](https://discuss.elastic.co/t/nonstop-lumberjack-pipeline-is-blocked-messages/25246/9 "2015-09-10T16:21:34Z")

</div>

I was able to resolve this by removing the multiline filter.

---

<div class="post-metadata">

**Author:** ![D\_Luffy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d_luffy/32/4687_2.png) [@D\_Luffy](https://discuss.elastic.co/u/D_Luffy)\
**Post date:** [September 10, 2015, 11:22pm UTC](https://discuss.elastic.co/t/nonstop-lumberjack-pipeline-is-blocked-messages/25246/10 "2015-09-10T23:22:42Z")

</div>

So far its holding up with some additional hardware but I presume it's just a matter of time to clog the pipeline.

I am considering topology changes,

currently the flow is similar to:  
shipper(s) lumberjack + tcp -\> lohstash indexer (x2 auto scaled upto 4) -\> Elasticsearch data nodes X2

considering  
shipper(s) lumberjack + tcp -\> logstash -\> Redis -\> Logstash (all filtering happens here) -\> elasticsearch

this adds an additional layer but hope it will be fast enough to accept all incoming requests.

All in all the Ideal would be to get the back pressure working across all inputs consistently.

---

<div class="post-metadata">

**Author:** ![Edrees](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/edrees/32/6685_2.png) [@Edrees](https://discuss.elastic.co/u/Edrees)\
**Post date:** [December 17, 2015, 5:57am UTC](https://discuss.elastic.co/t/nonstop-lumberjack-pipeline-is-blocked-messages/25246/11 "2015-12-17T05:57:39Z")

</div>

Me too was getting lot of this error in my logstash.log by changing the lumberjack ssl port I resolve the issue now my logs are coming nicely

---

<div class="post-metadata">

**Author:** ![plumpNation](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/plumpnation/32/7637_2.png) [@plumpNation](https://discuss.elastic.co/u/plumpNation)\
**Post date:** [February 9, 2016, 8:05am UTC](https://discuss.elastic.co/t/nonstop-lumberjack-pipeline-is-blocked-messages/25246/12 "2016-02-09T08:05:18Z")

</div>

I'm also experiencing this but figured it was simply poorly configured spool\_size and underpowered vms.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:12am UTC](https://discuss.elastic.co/t/nonstop-lumberjack-pipeline-is-blocked-messages/25246/13 "2017-07-06T05:12:37Z")

</div>


