Lumberjack Input Ceasing to Function


#1

Hello all!

I'm running several logstash-forwarder instances on remote servers that connect back to a central ELK machine. Events from all logstash-forwarder instances process fine for the majority of the time but seemingly at random logstash stops processing events and generates the following error:

{:timestamp=>"2015-08-17T04:02:43.983000-0600", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Lumberjack input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn}
{:timestamp=>"2015-08-17T04:02:43.984000-0600", :message=>"CircuitBreaker::Open", :name=>"Lumberjack input", :level=>:warn}
{:timestamp=>"2015-08-17T04:02:43.986000-0600", :message=>"Exception in lumberjack input thread", :exception=>#<LogStash::CircuitBreaker::OpenBreaker: for Lumberjack input>, :level=>:error}
{:timestamp=>"2015-08-17T04:03:21.181000-0600", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Lumberjack input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn}
{:timestamp=>"2015-08-17T04:03:21.184000-0600", :message=>"CircuitBreaker::Open", :name=>"Lumberjack input", :level=>:warn}

My logstash config is as follows:

input {
lumberjack {
port => 5043
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}

filter {
if "Connection from UDP:" in [message] {
drop {}
}
if "INPUT pkt died:" in [message] {
drop {}
}
if [file] == "/var/log/messages" {
grok {
match => [ "message", "%{SYSLOGTIMESTAMP:event_timestamp} %{WORD:hostname} %{GREEDYDATA:message_remainder}" ]
}
}
}

output {
redis {
data_type => "list"
key => "logstash"
}
stdout {}
if [file] == "/var/log/messages" {
file {
path => "/var/log/logstash/messages"
message_format => "%{event_timestamp} %{host} %{message_remainder}"
}
}
}

Any ideas as to why this happens? No other errors appear in logstash logs or in logstash-forwarder logs.


(Mark Walkom) #2

What versions are you on?


#3

Logstash - 1.5.2
Logstash-Forwarder - 0.3.1

CentOS 6 on all involved machines


(Taylor) #4

(Working along with davisv) We are also running elastic 1.7.1. Upgrading to logstash 1.5.3 did not help this issue.


(Randall Williams) #5

Seeing the same issue on 1.5.4 with lumberjack input plugin version 1.0.4.


(system) #6