Hi Ruflin,
Thanks for reply! Below is the log information of logstash. It saids 'the
pipeline is blocked.' and 'No space left on device'. We think it maybe
because that the filebeats had collected so many log files to the logstash
server that blocked the pipe line?
We have one logstash server running logstash and elasticsearch. We also
have many other compute nodes that filebeat will collect log files on.
{:timestamp=>"2016-07-22T17:20:22.610000-0400",
:message=>"CircuitBreaker::rescuing exceptions", :name=>"Beats input",
:exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong,
:level=>:warn}
{:timestamp=>"2016-07-22T17:20:22.622000-0400", :message=>"Beats input: The
circuit breaker has detected a slowdown or stall in the pipeline, the input
is closing the current connection and rejecting new connection until the
pipeline recover.",
:exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker,
:level=>:warn}
{:timestamp=>"2016-07-22T17:20:22.622000-0400", :message=>"Beats input: The
circuit breaker has detected a slowdown or stall in the pipeline, the input
is closing the current connection and rejecting new connection until the
pipeline recover.",
:exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker,
:level=>:warn}
{:timestamp=>"2016-07-22T17:20:23.070000-0400", :message=>"Beats input: the
pipeline is blocked, temporary refusing new connection.",
:reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2016-07-22T17:20:23.072000-0400",
:message=>"CircuitBreaker::Open", :name=>"Beats input", :level=>:warn}
{:timestamp=>"2016-07-22T17:20:23.073000-0400", :message=>"Beats input: The
circuit breaker has detected a slowdown or stall in the pipeline, the input
is closing the current connection and rejecting new connection until the
pipeline recover.",
:exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::OpenBreaker,
:level=>:warn}
{:timestamp=>"2016-07-22T17:20:23.573000-0400", :message=>"Beats input: the
pipeline is blocked, temporary refusing new connection.",
:reconnect_backoff_sleep=>0.5, :level=>:warn}
=>"node-32.moc.ne.edu", "tags"=>["beats_input_codec_plain_applied"]},
@metadata_accessors=#<LogStash::Util::Accessors:0x158c67ae
@store={"type"=>"log", "beat
"=>"filebeat"}, @lut={"[type]"=>[{"type"=>"log", "beat"=>"filebeat"},
"type"], "[beat]"=>[{"type"=>"log", "beat"=>"filebeat"}, "beat"]}>,
@cancelled=false>],
:response=>{"create"=>{"_index"=>"filebeat-2016.07.23", "_type"=>"log",
"_id"=>"AVYUet5p6ls66B28P6c0", "status"=>500,
"error"=>{"type"=>"exception", "reason
"=>"failed to sync translog", "caused_by"=>{"type"=>"i_o_exception",
"reason"=>"No space left on device"}}}}, :level=>:warn}