Hello all!
I'm running several logstash-forwarder instances on remote servers that connect back to a central ELK machine. Events from all logstash-forwarder instances process fine for the majority of the time but seemingly at random logstash stops processing events and generates the following error:
{:timestamp=>"2015-08-17T04:02:43.983000-0600", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Lumberjack input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn}
{:timestamp=>"2015-08-17T04:02:43.984000-0600", :message=>"CircuitBreaker::Open", :name=>"Lumberjack input", :level=>:warn}
{:timestamp=>"2015-08-17T04:02:43.986000-0600", :message=>"Exception in lumberjack input thread", :exception=>#<LogStash::CircuitBreaker::OpenBreaker: for Lumberjack input>, :level=>:error}
{:timestamp=>"2015-08-17T04:03:21.181000-0600", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Lumberjack input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn}
{:timestamp=>"2015-08-17T04:03:21.184000-0600", :message=>"CircuitBreaker::Open", :name=>"Lumberjack input", :level=>:warn}
My logstash config is as follows:
input {
lumberjack {
port => 5043
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
filter {
if "Connection from UDP:" in [message] {
drop {}
}
if "INPUT pkt died:" in [message] {
drop {}
}
if [file] == "/var/log/messages" {
grok {
match => [ "message", "%{SYSLOGTIMESTAMP:event_timestamp} %{WORD:hostname} %{GREEDYDATA:message_remainder}" ]
}
}
}
output {
redis {
data_type => "list"
key => "logstash"
}
stdout {}
if [file] == "/var/log/messages" {
file {
path => "/var/log/logstash/messages"
message_format => "%{event_timestamp} %{host} %{message_remainder}"
}
}
}
Any ideas as to why this happens? No other errors appear in logstash logs or in logstash-forwarder logs.