# Lumberjack input: the pipeline is blocked, temporary refusing new connection **URL:** \ **Category:** Logstash\ **Created:** [January 21, 2016, 11:23pm UTC](https://discuss.elastic.co/t/lumberjack-input-the-pipeline-is-blocked-temporary-refusing-new-connection/39829 "2016-01-21T23:23:58Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![daq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daq/32/33426_2.png) [@daq](https://discuss.elastic.co/u/daq)\ **Post date:** [January 21, 2016, 11:23pm UTC](https://discuss.elastic.co/t/lumberjack-input-the-pipeline-is-blocked-temporary-refusing-new-connection/39829/1 "2016-01-21T23:23:58Z")
We have a simple setup with 4 servers: Logstash (Log receiver) | / Redis (Queue) | / ElasticSearch (Storage) | / Kibana (Viewer) Receiver log is full of similar messages ``` {:timestamp=>"2016-01-21T15:17:49.408000-0800", :message=>"Lumberjack input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::CircuitBreaker::OpenBreaker, :level=>:warn} {:timestamp=>"2016-01-21T15:17:49.602000-0800", :message=>"Lumberjack input: the pipeline is blocked, temporary refusing new connection.", :level=>:warn} ``` There are no errors or significant cpu/memory/network load on any of the other servers. There is also no significant cpu/memory/network load on the receiving server. How do we identify the bottleneck? Thanks! ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 22, 2016, 6:57am UTC](https://discuss.elastic.co/t/lumberjack-input-the-pipeline-is-blocked-temporary-refusing-new-connection/39829/2 "2016-01-22T06:57:58Z")
Do you have some kind of plugin to Elasticsearch to pull messages from Redis? Or is your graph incomplete? Normally one would use Logstash between Redis and Elasticsearch. ---
**Author:** ![daq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daq/32/33426_2.png) [@daq](https://discuss.elastic.co/u/daq)\ **Post date:** [January 22, 2016, 8:04am UTC](https://discuss.elastic.co/t/lumberjack-input-the-pipeline-is-blocked-temporary-refusing-new-connection/39829/3 "2016-01-22T08:04:13Z")
Second Logstash server is running on the same instance as ES server. It pulls from Redis and outputs into ES. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 22, 2016, 8:20am UTC](https://discuss.elastic.co/t/lumberjack-input-the-pipeline-is-blocked-temporary-refusing-new-connection/39829/4 "2016-01-22T08:20:29Z")
The error message suggests that Logstash isn't able to push events to Redis as fast as events are arriving via the lumberjack protocol. Do you have any inflow to Redis? Is there anything interesting in the logs if you crank up the logging verbosity with `--verbose`? ---
**Author:** ![daq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daq/32/33426_2.png) [@daq](https://discuss.elastic.co/u/daq)\ **Post date:** [January 22, 2016, 7:46pm UTC](https://discuss.elastic.co/t/lumberjack-input-the-pipeline-is-blocked-temporary-refusing-new-connection/39829/5 "2016-01-22T19:46:18Z")
Redis server is largely idle. I've been monitoring network usage, CPU and RAM and there are no indication that it is under any load. I updated Redis logging level to verbose, but I'm not a 100% what to look for in Redis logs. They appear normal: > [13501] 22 Jan 11:41:41 \* DB saved on disk > [1332] 22 Jan 11:41:41 \* Background saving terminated with success > [1332] 22 Jan 11:42:55 \* 10000 changes in 60 seconds. Saving... > [1332] 22 Jan 11:42:55 \* Background saving started by pid 13502 > [13502] 22 Jan 11:42:55 \* DB saved on disk > [1332] 22 Jan 11:42:55 \* Background saving terminated with success > [1332] 22 Jan 11:44:08 \* 10000 changes in 60 seconds. Saving... > [1332] 22 Jan 11:44:08 \* Background saving started by pid 13507 > [13507] 22 Jan 11:44:08 \* DB saved on disk > [1332] 22 Jan 11:44:08 \* Background saving terminated with success > [1332] 22 Jan 11:45:19 \* 10000 changes in 60 seconds. Saving... > [1332] 22 Jan 11:45:19 \* Background saving started by pid 13510 > [13510] 22 Jan 11:45:19 \* DB saved on disk > [1332] 22 Jan 11:45:19 \* Background saving terminated with success ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 23, 2016, 4:20pm UTC](https://discuss.elastic.co/t/lumberjack-input-the-pipeline-is-blocked-temporary-refusing-new-connection/39829/6 "2016-01-23T16:20:44Z")
I was referring to the Logstash log. I'd be surprised if there wasn't some kind of indication of why it's slow or blocked. ---
**Author:** ![daq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daq/32/33426_2.png) [@daq](https://discuss.elastic.co/u/daq)\ **Post date:** [January 25, 2016, 11:57pm UTC](https://discuss.elastic.co/t/lumberjack-input-the-pipeline-is-blocked-temporary-refusing-new-connection/39829/7 "2016-01-25T23:57:41Z")
Before I got a chance to increase logging level on Logstash, the errors magically disappeared. Am I correct in saying that if we discover that Redis is actually overloaded, adding a second server is as simple as adding a second host to the output like so: ``` output { redis { data_type => "list" key => "logstash:cache:dev" host => ["redis1", "redis2"] } } ``` and then collecting from two servers like so: ``` input { redis { host => "redis1" data_type => "list" key => "logstash:cache:dev" } } input { redis { host => "redis2" data_type => "list" key => "logstash:cache:dev" } } ``` Thanks! ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 26, 2016, 6:57am UTC](https://discuss.elastic.co/t/lumberjack-input-the-pipeline-is-blocked-temporary-refusing-new-connection/39829/8 "2016-01-26T06:57:49Z")
I'm not completely sure the redis output falls back to the remaining servers if one of them pushes back because of load (if that indeed is the problem here). I'd inspect the code and try it out before concluding. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:14am UTC](https://discuss.elastic.co/t/lumberjack-input-the-pipeline-is-blocked-temporary-refusing-new-connection/39829/9 "2017-07-06T05:14:22Z")