The Pipeline is blocked

Elastic Stack Beats
1

Having some trouble resolving this issue on my own. Here is the error at the end of my logstash.log

{:timestamp=>"2016-07-11T14:20:06.631000-0400", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2016-07-11T14:20:06.951000-0400", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2016-07-11T14:20:07.452000-0400", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2016-07-11T14:20:07.953000-0400", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2016-07-11T14:20:08.454000-0400", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2016-07-11T14:20:08.955000-0400", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2016-07-11T14:20:09.460000-0400", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2016-07-11T14:20:09.961000-0400", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2016-07-11T14:20:10.462000-0400", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2016-07-11T14:20:10.963000-0400", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2016-07-11T14:20:11.464000-0400", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2016-07-11T14:20:11.968000-0400", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2016-07-11T14:20:12.469000-0400", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}

My logstash config files are:

02-beats-input.conf

input {
beats {
type => beats
port => 5044
congestion_threshold => "40"
}
}

10-syslog-filter.conf:

filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}

30-elasticsearch-output.conf:

output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
stdout { codec => rubydebug }
}

Please let me know if you need any more information. I saw that there was a similar post made to which adding the http protocol to the output plugin which actually caused my LS to stop working as far as I could tell.

2

It redid my formatting so just assume everything is tabbed correctly.

3

What kind of message rates are you seeing? Don't rule out the possibility that Elasticsearch isn't capable of accept the amount of messages you're trying to send to it.

I saw that there was a similar post made to which adding the http protocol to the output plugin which actually caused my LS to stop working as far as I could tell.

That advice was applicable for Logstash pre-2.0.

It redid my formatting so just assume everything is tabbed correctly.

If you format your configuration as code using the </> button you won't have that problem.

4

@mits This post should probably be moved into the logstash forum?

5

the beats input plugin in logstash has a circuit-breaker with default of 5-seconds. If logstash output can not deal with amount of data or filters are too slow (e.g. grok basically being a regex), the pipeline in logstash becomes congested and the circuit breaker will start closing connections.

Connection timeout in logstash is configured via congestion_threshold. Setting this to a very very large value (e.g. a few months), basically disables the circuit breaker in logstash. There is still a request timeout in beats reconnecting if logstash can not handle a batch of events in time. The default timeout is 30 seconds. Increase to whatever makes sense for you.

I see you've set congestion_threshold to 40 seconds already. How did you choose this value? Did you do any throughput testings with and without filters to get some baseline numbers?

6

This topic was automatically closed after 21 days. New replies are no longer allowed.