Just upgraded from Logstash 1.4.5 to 1.5.4 and have been beating my head against the wall trying to get lumberjack to work again. My issue is that if tcp/5000 is open (the port I have lumberjack listing on) then logstash will start but then crash a good 30 seconds later - usually right after startup. If I use iptables to drop all inbound packets on port 5000 then logstash will start, take inbound syslog and rabbitmq data, and run happily UNTIL I remove iptables INPUT rule and open the flood gates for lumberjack. When logstash dies, I get the error below in the logs. I've tried changing the USE_RUBY setting, upgraded from Java 7 to Java 8, and googled my heart out but find very little for "Mutex not locked" and "No message available" in conjunction with Concurrency. Any help would be greatly appreciated.
Error messages are below and system configuration is below that.
{:timestamp=>"2015-09-12T02:06:35.168000-0500", :message=>"Lumberjack input: unhandled exception", :exception=>#<ThreadError: Mutex is not locked>, :backtrace=>["org/jruby/ext/thread/Mutex.java:106:in `unlock'", "org/jruby/ext/thread/Mutex.java:151:in `synchronize'", "..."], :level=>:error}
{:timestamp=>"2015-09-12T02:33:58.067000-0500", :message=>"Lumberjack input: unhandled exception", :exception=>#<ConcurrencyError: No message available>, :backtrace=>["org/jruby/ext/thread/ConditionVariable.java:98:in `wait'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-lumberjack-1.0.5/lib/logstash/sized_queue_timeout.rb:30:in `push'", "org/jruby/ext/thread/Mutex.java:149:in `synchronize'",..."], :level=>:error}
Here are my system specs:
OS: Cent OS 7.1.1503 x64
RAM: 8GB
CPUS: 2
Java: Oracle java version "1.8.0_60"
System ruby: ruby 2.0.0p598 (2014-11-13) [x86_64-linux]
SELINUX: disabled
/etc/sysconfig/logstash:
LS_HEAP_SIZE="3096m"
LS_OPEN_FILES=16384
/etc/logstash/conf.d/logstash.conf
input {
rabbitmq {
host => "rmq01"
durable => true
key => "prod"
user => "logstash"
password => "password"
vhost => "/logs"
exchange => "logstash"
queue => "prod"
tags => "prod"
}
rabbitmq {
host => "rmq01"
durable => true
key => "train"
user => "logstash"
password => "password"
vhost => "/logs"
exchange => "logstash"
queue => "train"
tags => "train"
}
syslog {
port => 1514
type => "Syslog"
}
lumberjack {
port => 5000
ssl_certificate => "/opt/logstash/certs/logstash.crt"
ssl_key => "/opt/logstash/certs/logstash.key"
tags => "prod"
}
}
filter {
-- Filter definitions
}
output {
elasticsearch {
host => "elasticcluster01"
protocol => "transport"
cluster => "elasticsearch-01"
}
statsd {
type => "IISLog"
host => "statsd01"
port => 2003
sender => "%{hostname}"
increment => "%{statuscode}"
timing => [ "timetaken", "%{timetaken}" ]
}
}
--
Ed