Filebeat logstash primary shard error


(bala) #1

Hi All

I have logstash 2.4.0 and filebeat 5.2.2...filebeat reads log files and send it to logstash..logstash send it to our ELK cluster running 2.2.0

here is my filebeat config

========

filebeat.prospectors:

  • input_type: log

    Paths that should be crawled and fetched. Glob based paths.

    paths:

    • /apps/dashboard/flumedir1/cdr/*.COMPLETED
    • /apps/dashboard/flumedir2/cdr/*.COMPLETED
      ignore_older: 240h

output.logstash:

The Logstash hosts

hosts: ["localhost:9212"]
timeout: 320

=========================
here is my logstash

input {
beats {
codec => "json"
port => 9212
}
}
filter {
json {
source => "message"
}
ruby {
code => "require 'digest/md5';
event['@metadata']['cdr_id'] = Digest::MD5.hexdigest(event['X'] + event['I'] + event['LI']);
event['type'] = 'cgroup-' + (Digest::MD5.hexdigest(event['X']).to_i(16) % 1000).to_s"
}
date {
match => [ "ST", "UNIX_MS" ]
timezone => "UTC"
}
}
output {
if [X] in [""]
elasticsearch {
hosts => ["elk-node1:9200","elk-node2:9200"]
flush_size => 1000
document_id => "%{[@metadata][cdr_id]}"
index => "lob%{+YYYYMMdd}"
}
}
}

======
In my log files, I see below messages
{:timestamp=>"2017-07-26T00:05:59.762000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Beats input", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn}
{:timestamp=>"2017-07-26T00:05:59.763000+0000", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2017-07-26T00:06:08.291000+0000", :message=>"retrying failed action with response code: 503 ({"type"=>"unavailable_shards_exception", "reason"=>"[lob20170723][1] primary shard is not active Timeout: [1m], request: [shard bulk {[lob20170723][1]}]"})", :level=>:info}
{:timestamp=>"2017-07-26T00:07:04.768000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Beats input", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn}
{:timestamp=>"2017-07-26T00:07:04.770000+0000", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}

lob20170723 is one of our index name...
I checked ELK cluster health ., and no issues with it., (green state and all shards are in STARTED state and also file system space usage is less than 50%)..

Any suggestions/recommendations to fix this error ?

Shankar


(Mark Walkom) #2

Can you please reformat your post and use code tags to make it easier to read..


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.