Hello,
My program will frequently modify the configure of syslog output plugin,
the host may be reachable or unreachable.
I try many times,and it works well,but randomly the log threw out stalling_threads_info.
At the same time forwarding also fail,the configure and error message are below.
It seems like the worker encounter some race conditions so the thread stall?
I found the pattern that cause this issue:
after i configure the unreachable tcp host,the error message shows syslog tcp output exception.
Then the stalling thread info logs shows up!
logstash environment parameters
config.reload.automatic=true
config.reload.interval=5
Conguration:
output {
syslog {
appname => "%{appname}"
facility => "%{facility}"
severity => "%{severity}"
host => "192.168.2.2"
port => 3000
protocol => "tcp"
}
syslog {
appname => "%{appname}"
facility => "%{facility}"
severity => "%{severity}"
host => "192.168.1.1"
port => 514
protocol => "udp"
}
}
TCP WARN logs:
syslog tcp output exception: closing, reconnecting and resending event {:host=>"10.0.0.3", :port=>5, :exception=>#<Errno::ECONNREFUSED: Connection refused - connect(2) for "10.0.0.3" port 5>, :backtrace=>["org/jruby/ext/socket/RubyTCPSocket.java:134:in initialize'", "org/jruby/RubyIO.java:875:innew'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:209:in connect'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:177:inpublish'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-plain-3.0.6/lib/logstash/codecs/plain.rb:40:in encode'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:147:inreceive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in block in multi_receive'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:114:inmulti_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:97:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:373:inblock in output_batch'", "org/jruby/RubyHash.java:1343:in each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:372:inoutput_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:324:in worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:287:inblock in start_workers'"], :event=>#LogStash::Event:0x67d24825}
WARN logs:
[2019-04-03T00:54:13,593][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>126, "name"=>"[main]<syslog", "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-input-syslog-3.4.1/lib/logstash/inputs/syslog.rb:119:in join'"}], ["LogStash::Filters::Mutate", {"update"=>{"program"=>"aaa"}, "id"=>"3db2c3ef3a506ad16d0b96c0be1c43fbe6084c63c77b97a1ecd4e6af174ac6fe"}]=>[{"thread_id"=>121, "name"=>"[main]>worker0", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:373:inmulti_receive'"}, {"thread_id"=>122, "name"=>"[main]>worker1", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:373:in multi_receive'"}, {"thread_id"=>123, "name"=>"[main]>worker2", "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:209:innew'"}, {"thread_id"=>124, "name"=>"[main]>worker3", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:373:in `multi_receive'"}]}}