Greetings
I am using centrally managed pipelines, so that I can change the pipeline in all my logstash nodes at once. The configuration is following:
xpack.management.enabled: true
xpack.management.elasticsearch.url: ["https://x.x.x.x:9200", "https://x.x.x.x:9200", ...]
xpack.management.elasticsearch.username: "logstash_admin"
xpack.management.elasticsearch.password: "xxxxx"
xpack.management.logstash.poll_interval: 5s
xpack.management.pipeline.id: ["clog"]
xpack.management.elasticsearch.ssl.ca: /etc/logstash/certs/pemc-clog-ca.crt
However, when I make edits to the pipeline in Kibana and click the [Save and deploy] button, I get this error message in the logs, repeated indefinitely:
[2018-09-17T12:02:40,645][ERROR][org.logstash.execution.AbstractPipelineExt] Logstash failed to create queue.
org.logstash.LockException: The queue failed to obtain exclusive access, cause: Lock held by this virtual machine on lock path: /var/lib/logstash/queue/clog/.lock
at org.logstash.ackedqueue.Queue.open(Queue.java:262) ~[logstash-core.jar:?]
at org.logstash.ackedqueue.ext.JRubyAckedQueueExt.open(JRubyAckedQueueExt.java:97) ~[logstash-core.jar:?]
at org.logstash.ackedqueue.ext.JRubyWrappedAckedQueueExt.initialize(JRubyWrappedAckedQueueExt.java:39) ~[logstash-core.jar:?]
at org.logstash.ackedqueue.QueueFactoryExt.create(QueueFactoryExt.java:37) ~[logstash-core.jar:?]
at org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:130) [logstash-core.jar:?]
at org.logstash.execution.AbstractPipelineExt$INVOKER$i$3$0$initialize.call(AbstractPipelineExt$INVOKER$i$3$0$initialize.gen) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodThree.call(JavaMethod.java:1186) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:743) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:983) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:69) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:348) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:173) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:177) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.RubyClass.newInstance(RubyClass.java:1001) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:89) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:214) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:200) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:358) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:195) [jruby-complete-9.1.13.0.jar:?]
at usr.share.logstash.logstash_minus_core.lib.logstash.agent.invokeOther13:execute(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:309) [jruby-complete-9.1.13.0.jar:?]
at usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:309) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.runtime.Block.call(Block.java:124) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.RubyProc.call(RubyProc.java:289) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.RubyProc.call(RubyProc.java:246) [jruby-complete-9.1.13.0.jar:?]
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104) [jruby-complete-9.1.13.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
The logstash keeps running the old pipeline, and the new one doesn't get loaded unless I restart logstash. This started happening after I upgraded to 6.4.0 from 6.3.2, but I didn't see anything relevant in the Breaking Changes section.
Is there any solution to this?