I think I have a problem with log4j input plugin.
OS: Scientific Linux CERN SLC release 6.6 (Carbon)
Logstash: 2.3.4
Config pipeline.conf:
input {
log4j {
mode => "client"
host => log_output_host
port => 26021
}
}
output {
stdout {codec => rubydebug}
}
Then I do:
./bin/logstash -f pipeline.conf
It starts OK, I see nice output for around 10 minutes, then it stops. CPU usage for logstash drops from 2-7% to 0%. I know that log4j socket hub output (on log_output_host:26021) is fine because if I start another instance of logstash with the same config, it starts and outputs receiving events.
When it is stuck and I hit Ctrl+C, I get:
SIGINT received. Shutting down the agent. {:level=>:warn}
stopping pipeline {:id=>"main"}
Closing inputs {:level=>:info}
Closed inputs {:level=>:info}
Received shutdown signal, but pipeline is still waiting for in-flight events
to be processed. Sending another ^C will force quit Logstash, but this may cause
data loss. {:level=>:warn}
Then after some time:
{"inflight_count"=>0, "stalling_thread_info"=>{"other"=>[{"thread_id"=>17, "name"=>"[main]<log4j", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/logstash-input-log4j-2.0.7-java/lib/logstash/inputs/log4j.rb:105:in `read_object'"}, {"thread_id"=>18, "name"=>"[main]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:309:in `synchronize'"}, {"thread_id"=>19, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:309:in `synchronize'"}, {"thread_id"=>20, "name"=>"[main]>worker2", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:218:in `synchronize'"}, {"thread_id"=>21, "name"=>"[main]>worker3", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:218:in `synchronize'"}, {"thread_id"=>22, "name"=>"[main]>worker4", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:218:in `synchronize'"}, {"thread_id"=>23, "name"=>"[main]>worker5", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:218:in `synchronize'"}, {"thread_id"=>24, "name"=>"[main]>worker6", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:309:in `synchronize'"}, {"thread_id"=>25, "name"=>"[main]>worker7", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:218:in `synchronize'"}, {"thread_id"=>26, "name"=>"[main]>worker8", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:218:in `synchronize'"}, {"thread_id"=>27, "name"=>"[main]>worker9", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:218:in `synchronize'"}, {"thread_id"=>28, "name"=>"[main]>worker10", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:309:in `synchronize'"}, {"thread_id"=>29, "name"=>"[main]>worker11", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:309:in `synchronize'"}]}} {:level=>:warn}
The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information. {:level=>:error}
Does anybody have any advice?