we have a logstash setup with mainly input from syslog, grok filter and output to elasticsearch, and after a while no new message appear in ES and rsyslog starts to queue up events, suggesting that logstash is hanging somewhere.
the only way to resolve this for now is to restart logstash, and then the following message appears in the logstash-plain.log:
[2017-01-24T11:05:50,896][WARN ][logstash.shutdownwatcher ] {}
[2017-01-24T11:05:50,899][ERROR][logstash.shutdownwatcher ] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.
the logstash-plain.log has an error now and then with an unreadable stacktrace, see below
no messages in the elasticsearch log either.
we would like to get to the bottom of this (we suspect some log messages make the grok filter hang).
so we have the following questions:
what logs is being referred to?
is there a way to dump the events in the logstash queue and/or those being processed by the filter.
is there a way to timeout blocked/busy plugins and have them skip whatever they are doing instead of blocking entire logstash?
many thanks
stijn
first bytes of the stacktrace (total is bigger than the allowed message limit here)
[2017-01-24T10:57:03,464][ERROR][logstash.pipeline ] .call(CachingCallSite.java:168)", "org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)", "org.jruby.ast.DAsgnNode.interpret(DAsgnNode.java:110)", "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)", "org.jruby.ast.IfNode.interpret(IfNode.java:116)", "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)", "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)", "org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)", "org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)", "org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)", "org.jruby.runtime.Block.yield(Block.java:142)", "org.jruby.RubyArray.eachCommon(RubyArray.java:1606)", "org.jruby.RubyArray.each(RubyArray.java:1613)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)", "org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)", "org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)", "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)", "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)",