# Pipeline is full, she can't take much more captain!

**URL:** <https://discuss.elastic.co/t/pipeline-is-full-she-cant-take-much-more-captain/67536>\
**Category:** Beats\
**Tags:** winlogbeat\
**Created:** [November 29, 2016, 7:24pm UTC](https://discuss.elastic.co/t/pipeline-is-full-she-cant-take-much-more-captain/67536 "2016-11-29T19:24:48Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![gchampion](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gchampion](https://discuss.elastic.co/u/gchampion)\
**Post date:** [November 29, 2016, 7:24pm UTC](https://discuss.elastic.co/t/pipeline-is-full-she-cant-take-much-more-captain/67536/1 "2016-11-29T19:24:48Z")

</div>

Everything was going fine until I doubled the amount of log forwarders. I went from around 5,000/minute to 5/minute (not kidding). I've since cut the amount of log forwarders down to the original amount however I can't seem to get the performance back.

Logstash.log:  
{:timestamp=\>"2016-11-29T13:56:27.261000-0500", :message=\>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect\_backoff\_sleep=\>0.5, :level=\>:warn}  
{:timestamp=\>"2016-11-29T13:56:27.761000-0500", :message=\>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect\_backoff\_sleep=\>0.5, :level=\>:warn}  
{:timestamp=\>"2016-11-29T13:56:28.262000-0500", :message=\>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect\_backoff\_sleep=\>0.5, :level=\>:warn}

Logstash: /etc/logstash/conf.d/02-beats-input.conf  
input {  
beats {  
port =\> 5044  
ssl =\> true  
ssl\_certificate =\> "/etc/pki/tls/certs/logstash-forwarder.crt"  
ssl\_key =\> "/etc/pki/tls/private/logstash-forwarder.key"  
congestion\_threshold =\> "40"  
}  
}

I maybe gettting the above confused with , Possibly I'm changing my congestion settings in the wrong place:  
/opt/logstash/bin/logstash.conf  
Input {  
beats {  
port =\> 5044  
}  
}

output {  
elasticsearch {  
hosts =\> "localhost:9200"  
manage\_template =\> false  
index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}"  
document\_type =\> "%{[@metadata][type]}"  
}  
}

Stats on the box look fine (CPU / Memory)

plugin:elasticsearch Request Timeout after 1500ms will also display when kibana is starting from time to time.

---

<div class="post-metadata">

**Author:** ![gchampion](https://avatars.discourse-cdn.com/v4/letter/g/e79b87/32.png) [@gchampion](https://discuss.elastic.co/u/gchampion)\
**Post date:** [November 29, 2016, 9:25pm UTC](https://discuss.elastic.co/t/pipeline-is-full-she-cant-take-much-more-captain/67536/2 "2016-11-29T21:25:58Z")

</div>

Hey PJ, Let me help you with this, what's the status of your elasticsearch service?

Funny you should ask it says out of memory, however I cant see to change the heap space.

Here I found a snippet that might help, seems the spot to change memory has changed.

> <https://stackoverflow.com/questions/18132719/how-to-change-elasticsearch-max-memory-size>

\*Marked as self resolved.

PJ is a smart awesome guy who needs to pay more attention to Java, like always.

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [November 29, 2016, 9:41pm UTC](https://discuss.elastic.co/t/pipeline-is-full-she-cant-take-much-more-captain/67536/3 "2016-11-29T21:41:07Z")

</div>

@gchampion Glad you found the issue. As you add more nodes you might want to walk through the LS [performance troubleshooting](https://www.elastic.co/guide/en/logstash/current/performance-troubleshooting.html) checklist to help optimize Logstash.

What version of LS are you running? `congestion_threshold` is being deprecated in LS 5.0 because it's not needed as the protocol between LS and Beats handles the back-pressure.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 27, 2016, 9:41pm UTC](https://discuss.elastic.co/t/pipeline-is-full-she-cant-take-much-more-captain/67536/4 "2016-12-27T21:41:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.