Logstash circuit breaker triggered but marvel monitoring metrics seems OK


(Maxhyjal) #1

I've run into this prior to 2.0 and after I upgrade the Elasticsearch and Logstash stack to 2.0, is there a good way to get around this?

Keep getting those in logstash.log
message=>"Lumberjack input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2015-11-19T21:49:50.388000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Lumberjack input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn}
{:timestamp=>"2015-11-19T21:49:50.388000+0000", :message=>"Lumberjack input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2015-11-19T21:49:50.718000+0000", :message=>"Lumberjack input: the pipeline is blocked, temporary refusing new connection.", :level=>:warn}


(Mark Walkom) #2

This happens because downstream is overloaded.
Are you monitoring your stack?


(system) #3