Hello, I am trying to add Persistent queues to logstash, however i am getting the below error. And i not sure how it resolve it. I am not sure if issue with config or logstash itself.
What i have done:
I have stopped logstash and removed .lock, same error
I have stopped logstash and removed the whole main queue dir, same error
I have remove the queue settings and logstash starts.
# node.name: test
path.data: /var/lib/logstash
config.reload.automatic: false
queue.type: persisted
queue.drain: true
queue.page_capacity: 64mb
queue.max_events: 0
queue.max_bytes: 1024mb
queue.checkpoint.acks: 512
queue.checkpoint.writes: 512
queue.checkpoint.interval: 512
log.level: info
path.logs: /var/log/logstash
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: ["http://172.16.2.128:9200"]
xpack.monitoring.elasticsearch.sniffing: true
xpack.monitoring.collection.interval: 10s
xpack.monitoring.collection.pipeline.details.enabled: true
Error Msg:
[2019-05-14T16:46:27,605][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, :thread=>"#<Thread:0x787f11a0 run>"}
[2019-05-14T16:46:27,682][ERROR][org.logstash.execution.AbstractPipelineExt] Logstash failed to create queue.
org.logstash.LockException: The queue failed to obtain exclusive access, cause: Lock held by this virtual machine on lock path: /var/lib/logstash/queue/main/.lock
at org.logstash.ackedqueue.Queue.open(Queue.java:263) ~[logstash-core.jar:?]
at org.logstash.ackedqueue.ext.JRubyAckedQueueExt.open(JRubyAckedQueueExt.java:101) ~[logstash-core.jar:?]
at org.logstash.ackedqueue.ext.JRubyWrappedAckedQueueExt.initialize(JRubyWrappedAckedQueueExt.java:42) ~[logstash-core.jar:?]
at org.logstash.ackedqueue.QueueFactoryExt.create(QueueFactoryExt.java:39) ~[logstash-core.jar:?]
at org.logstash.execution.AbstractPipelineExt.openQueue(AbstractPipelineExt.java:169) [logstash-core.jar:?]
at org.logstash.execution.AbstractPipelineExt$INVOKER$i$0$0$openQueue.call(AbstractPipelineExt$INVOKER$i$0$0$openQueue.gen) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:831) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183) [jruby-complete-9.2.7.0.jar:?]
at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183) [jruby-complete-9.2.7.0.jar:?]
at usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36) [jruby-complete-9.2.7.0.jar:?]
at usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.runtime.Block.call(Block.java:124) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.RubyProc.call(RubyProc.java:295) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.RubyProc.call(RubyProc.java:274) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.RubyProc.call(RubyProc.java:270) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105) [jruby-complete-9.2.7.0.jar:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-05-14T16:46:27,744][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"org.logstash.LockException: The queue failed to obtain exclusive access, cause: Lock held by this virtual machine on lock path: /var/lib/logstash/queue/main/.lock", :backtrace=>["org.logstash.execution.AbstractPipelineExt.openQueue(AbstractPipelineExt.java:172)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$0$0$openQueue.call(AbstractPipelineExt$INVOKER$i$0$0$openQueue.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:831)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:295)", "org.jruby.RubyProc.call(RubyProc.java:274)", "org.jruby.RubyProc.call(RubyProc.java:270)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:835)"]}