Today, I find out, my queue is full, and my logstash suddenly generate this error:
[2018-08-31T12:56:13,494][ERROR][logstash.pipeline ] Logstash failed to create queue {:pipeline_id=>"default-pipeline", "exception"=>"org.logstash.LockException: The queue failed to obtain exclusive access, cause: Lock held by this virtual machine on lock path: /var/lib/logstash/queue/default-pipeline/.lock", "backtrace"=>["org/logstash/ackedqueue/Queue.java:258:in `open'", "org/logstash/ackedqueue/ext/AbstractJRubyQueue.java:88:in `open'", "org/logstash/ackedqueue/ext/AbstractJRubyQueue$INVOKER$i$0$0$ruby_open.gen:-1:in `call'", "usr/share/logstash/logstash_minus_core/lib/logstash/util//usr/share/logstash/logstash-core/lib/logstash/util/wrapped_acked_queue.rb:39:in `invokeOther2:open'", "/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_acked_queue.rb:39:in `with_queue'", "usr/share/logstash/logstash_minus_core/lib/logstash/util//usr/share/logstash/logstash-core/lib/logstash/util/wrapped_acked_queue.rb:28:in `invokeOther2:with_queue'", "/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_acked_queue.rb:28:in `create_file_based'", "usr/share/logstash/logstash_minus_core/lib/logstash/util//usr/share/logstash/logstash-core/lib/logstash/util/wrapped_acked_queue.rb:-1:in `RUBY$method$create_file_based$0$__VARARGS__'", "usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/queue_factory.rb:29:in `invokeOther37:create_file_based'", "/usr/share/logstash/logstash-core/lib/logstash/queue_factory.rb:29:in `create'", "usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:172:in `invokeOther19:create'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:172:in `initialize'", "org/jruby/RubyClass.java:1015:in `newInstance'", "org/jruby/RubyClass$INVOKER$i$newInstance.gen:-1:in `call'", "usr/share/logstash/logstash_minus_core/lib/logstash/pipeline_action//usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `invokeOther9:new'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in `invokeOther5:execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in `block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in `invokeOther34:with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in `block in converge_state'", "org/jruby/RubyArray.java:1734:in `each'", "org/jruby/RubyArray$INVOKER$i$0$0$each.gen:-1:in `call'", "usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in `invokeOther56:each'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in `converge_state'", "usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in `invokeOther2:converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in `block in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in `invokeOther32:with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in `converge_state_and_update'", "usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/agent.rb:105:in `invokeOther1:converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:105:in `block in execute'", "org/jruby/RubyProc.java:289:in `call'", "org/jruby/RubyProc.java:273:in `call19'", "org/jruby/RubyProc$INVOKER$i$0$0$call19.gen:-1:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:18:in `interval'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:94:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:in `block in execute'", "org/jruby/RubyProc.java:289:in `call'", "org/jruby/RubyProc.java:273:in `call19'", "org/jruby/RubyProc$INVOKER$i$0$0$call19.gen:-1:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'", "org/jruby/RubyProc.java:289:in `call'", "org/jruby/RubyProc.java:246:in `call'", "java/lang/Thread.java:748:in `run'"]}
According to first line:
"exception"=>"org.logstash.LockException: The queue failed to obtain exclusive access, cause: Lock held by this virtual machine on lock path: /var/lib/logstash/queue/default-pipeline/.lock",
It looks like problem is that there is .lock file already. So I stopped logstash, remove .lock and start it again. The problem persist. I tryied to start pipeline in new directory, it works fine. Could anybody help me to figure out where is the problem and how to solve it? I have a few gigabites of logs there and cant send it to ES. I used my configuration for more than 2 months and this problem appear suddenly today. Could my pages file in queue be corrupted somehow?
There is also one more line, but the logs are too long, so I adding just snippet of it:
[2018-08-31T12:56:13,495][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:default-pipeline, :exception=>"IOError", :message=>"org.logstash.LockException: The queue failed to obtain exclusive access, cause: Lock held by this virtual machine on lock path: /var/lib/logstash/queue/default-pipeline/.lock"
Logstash version 6.2.2
UPDATE:
It looks like some file from queue was corrupted. Could not find solution but remove data from queue solve the issue.