Hi All
I have logstash 2.4.0 and filebeat 5.2.2...filebeat reads log files and send it to logstash..logstash send it to our ELK cluster running 2.2.0
here is my filebeat config
========
filebeat.prospectors:
-
input_type: log
Paths that should be crawled and fetched. Glob based paths.
paths:
- /apps/dashboard/flumedir1/cdr/*.COMPLETED
- /apps/dashboard/flumedir2/cdr/*.COMPLETED
ignore_older: 240h
output.logstash:
The Logstash hosts
hosts: ["localhost:9212"]
timeout: 320
=========================
here is my logstash
input {
beats {
codec => "json"
port => 9212
}
}
filter {
json {
source => "message"
}
ruby {
code => "require 'digest/md5';
event['@metadata']['cdr_id'] = Digest::MD5.hexdigest(event['X'] + event['I'] + event['LI']);
event['type'] = 'cgroup-' + (Digest::MD5.hexdigest(event['X']).to_i(16) % 1000).to_s"
}
date {
match => [ "ST", "UNIX_MS" ]
timezone => "UTC"
}
}
output {
if [X] in [""]
elasticsearch {
hosts => ["elk-node1:9200","elk-node2:9200"]
flush_size => 1000
document_id => "%{[@metadata][cdr_id]}"
index => "lob%{+YYYYMMdd}"
}
}
}
======
In my log files, I see below messages
{:timestamp=>"2017-07-26T00:05:59.762000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Beats input", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn}
{:timestamp=>"2017-07-26T00:05:59.763000+0000", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2017-07-26T00:06:08.291000+0000", :message=>"retrying failed action with response code: 503 ({"type"=>"unavailable_shards_exception", "reason"=>"[lob20170723][1] primary shard is not active Timeout: [1m], request: [shard bulk {[lob20170723][1]}]"})", :level=>:info}
{:timestamp=>"2017-07-26T00:07:04.768000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Beats input", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn}
{:timestamp=>"2017-07-26T00:07:04.770000+0000", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
lob20170723 is one of our index name...
I checked ELK cluster health ., and no issues with it., (green state and all shards are in STARTED state and also file system space usage is less than 50%)..
Any suggestions/recommendations to fix this error ?
Shankar