Any trick to avoid "Beats input: the pipeline is blocked, temporary refusing new connection."?

Elastic Stack Logstash
1

Hi,

I see a few traces (in logstash.log) like below and wonder if there is a way to configure them away.
The input to logstash from filebeat (not on same machines) should not cause congestion 383KBits/s (pure text strings).
Can I somehow see how much jam there is in the system?
Does this trace mean that events are discared or does logstash tell filebeat to wait sending data?
( I miss plenty of messages in Kibana)
Is it possible to see in any log for how long the block is?
I see no indication on filebeat machine that there is a problem?
Will filebeat log anything when is fails to send data?
Is it possible to configure filebeat and logstash better to avoid this problem?

{:timestamp=>"2016-04-25T08:15:33.983000+0000", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2016-04-25T08:15:34.483000+0000", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2016-04-25T08:15:34.984000+0000", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2016-04-25T08:15:35.484000+0000", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2016-04-25T08:15:35.985000+0000", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2016-04-25T08:15:36.485000+0000", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2016-04-25T08:15:36.985000+0000", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2016-04-25T08:15:42.530000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Beats input", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn}
{:timestamp=>"2016-04-25T08:15:42.539000+0000", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2016-04-25T08:15:46.042000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Beats input", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn}
{:timestamp=>"2016-04-25T08:15:46.042000+0000", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2016-04-25T08:15:50.044000+0000", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2016-04-25T08:15:50.045000+0000", :message=>"CircuitBreaker::Open", :name=>"Beats input", :level=>:warn}
{:timestamp=>"2016-04-25T08:15:50.045000+0000", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::OpenBreaker, :level=>:warn}
{:timestamp=>"2016-04-25T08:15:50.545000+0000", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2016-04-25T08:15:51.046000+0000", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2016-04-25T08:15:51.546000+0000", :message=>"Beats input: the pipeline is block

Br Mathias

2

Is there really nothing else in the log that indicates why the pipeline is clogged? What kind of outputs do you have?

3

Hi,

What else should I be looking for?
I am quite new to ELK and beats but I would expect it to work or not work when I start up. Not partly work.

I have restarted logstash/elasticsearch/nginx and kibana a couple of minutes ago and I still see the same problem after a while.

{:timestamp=>"2016-04-25T18:09:29.839000+0000", :message=>"SIGTERM received. Shutting down the pipeline.", :level=>:warn}
{:timestamp=>"2016-04-25T18:09:59.527000+0000", :message=>"Beats input: SSL Certificate will not be used", :level=>:warn}
{:timestamp=>"2016-04-25T18:09:59.528000+0000", :message=>"Beats input: SSL Key will not be used", :level=>:warn}
{:timestamp=>"2016-04-25T18:10:01.623000+0000", :message=>"You are using a deprecated config setting "message_format" set in file. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. You can achieve the same behavior with the 'line' codec If you have any questions about this, please visit the #logstash channel on freenode irc.", :name=>"message_format", :plugin=><LogStash::Outputs::File message_format=>"%{[message]}", path=>"/tmp/%{host}/%{type}-%{+YYYY-MM-dd}.log">, :level=>:warn}
{:timestamp=>"2016-04-25T18:10:08.147000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Beats input", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn}
{:timestamp=>"2016-04-25T18:10:08.150000+0000", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2016-04-25T18:10:17.163000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Beats input", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn}
{:timestamp=>"2016-04-25T18:10:17.164000+0000", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2016-04-25T18:30:27.414000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Beats input", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn}
{:timestamp=>"2016-04-25T18:30:27.415000+0000", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2016-04-25T18:30:27.447000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Beats input", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn}
{:timestamp=>"2016-04-25T18:30:27.447000+0000", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2016-04-25T18:31:25.279000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Beats input", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn}
{:timestamp=>"2016-04-25T18:31:25.280000+0000", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2016-04-25T18:31:26.283000+0000", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2016-04-25T18:31:26.287000+0000", :message=>"CircuitBreaker::Open", :name=>"Beats input", :level=>:warn}
{:timestamp=>"2016-04-25T18:31:26.287000+0000", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::OpenBreaker, :level=>:warn}
{:timestamp=>"2016-04-25T18:31:26.785000+0000", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2016-04-25T18:31:27.286000+0000", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}

4

And what kind of outputs do you have?

What else should I be looking for?

Anything that would indicate that something's wrong. Pipeline stalls should be easy to find, but you might want to increase the logging verbosity by starting Logstash with --verbose or even --debug (although with the latter you'll probably have problems separating the garbage from the potentially interesting messages).

5

I separated logstash and elasticsearch to run on different virtual machines and there are no new pipeline events generated in the logstash log.
Unfortunately did the main problem not disappear. I still loose data after letting filebeat send data to logstash.
I thought the blocked pipeline could explain why data is lost but It can not. I have to look at what is coming out from filebeat.

Br Mathias

6

Hi all,

I had the same issue today and I still don't really know why.

Maybe my reflexion could help :
I thought it could be a problem of memory usage, so I have upgraded the VM RAM from 2GB to 8GB.
Plus I've set the parameter LS_HEAP_SIZE="4g" in the file /opt/logstash/bin/logstash
Restarted Logstash and ElasticSearch just in case (but this is a Logstash problem).

Then I wised up to the fact that I had 2 outputs in my logstash.conf, 1 to ElasticSearch, the 2nd to another server by TCP ! BUT this server was shutdown, and I think logstash couldn't create the TCP connexion, raising this error :
message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}

Commenting this 2nd output in logstash.conf solved the problem.

Don't you have an output that is not up and running ?

7

Hi,

I tried the same thing but could only get 2 CPUs and 4Gb of memory.
Data flow better until I started to add GROK and Date filters. This slowed down logstash a lot. Hopefully it is fast enough on real machines.
I do not see the errors so original problem is solved.

Br Mathias

8

I have the same problem when i add the grok and date filter and the data flow is not really high, less than 100 messages per second. I have also split each service into different VM. Do you know by change how to solve it?

9

the problem cause by the grok and make the cup 100%
i have the same problem.but i cant resolve it but delete the grok