Logstash 6.4.0 reload config - queue failed to obtain exclusive access

Elastic Stack Logstash
1

I have Logstash 6.4.0 running with one persistent queue "main" and 8 pipeline workers. When sending SIGHUP I get an error. Restarting the service works fine.

[2018-08-29T08:38:51,873][WARN ][logstash.runner          ] SIGHUP received.
[2018-08-29T08:38:51,902][ERROR][org.logstash.execution.AbstractPipelineExt] Logstash failed to create queue.
org.logstash.LockException: The queue failed to obtain exclusive access, cause: Lock held by this virtual machine on lock path: /var/lib/logstash/queue/main/.lock
	at org.logstash.ackedqueue.Queue.open(Queue.java:262) ~[logstash-core.jar:?]
	at org.logstash.ackedqueue.ext.JRubyAckedQueueExt.open(JRubyAckedQueueExt.java:97) ~[logstash-core.jar:?]
	at org.logstash.ackedqueue.ext.JRubyWrappedAckedQueueExt.initialize(JRubyWrappedAckedQueueExt.java:39) ~[logstash-core.jar:?]
	at org.logstash.ackedqueue.QueueFactoryExt.create(QueueFactoryExt.java:37) ~[logstash-core.jar:?]
	at org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:130) [logstash-core.jar:?]
	at org.logstash.execution.AbstractPipelineExt$INVOKER$i$3$0$initialize.call(AbstractPipelineExt$INVOKER$i$3$0$initialize.gen) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodThree.call(JavaMethod.java:1186) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:743) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:983) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:69) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:348) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:173) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:177) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.RubyClass.newInstance(RubyClass.java:1001) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:89) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:214) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:200) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:358) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:195) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:323) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.runtime.Block.call(Block.java:124) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.RubyProc.call(RubyProc.java:289) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.RubyProc.call(RubyProc.java:246) [jruby-complete-9.1.13.0.jar:?]
	at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104) [jruby-complete-9.1.13.0.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
2

I experience the exact same thing after doing "yum upgrade logstash" to 6.4.0.
Same stacktrace as above.

"sudo kill 1 " does not work, but stopping and starting does.

It worked fine on logstash 6.3.2

3

I can conrfirm this.
Is there an issue already open?

4

I think this is supposed to be fixed in logstash 6.4.1.

https://www.elastic.co/guide/en/logstash/6.4/logstash-6-4-1.html

Pipeline reloading breaks with PQ enabled (#9986).

However, trying it out, I now see the behaviour that logstash prints
[2018-09-25T10:49:52,588][WARN ][logstash.runner ] SIGHUP received.

but nothing appears to happen! No error message as before, but no logging of pipeline reloading either.

Seems like we still have to stop/start logstash to update pipelines.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.