Logstash stopped running

Elastic Stack Logstash
1

I have a cloud elastic instance and a local VM running logstash that is reading data from RabbitMQ which is also local.

This has been running fine for 4 days then today I nocitce that the queue on RabbitMQ is growing as it has no consumer.

The Logstash servcie is running but I'm not seeing the kind of system load it generated previously.

I reviewed the logs and found the first error last night at 23:33

[2018-04-30T23:33:08,764][ERROR][logstash.pipeline        ] Logstash failed to create queue {:pipeline_id=>"ApiUsage", "exception"=>"java.nio.file.NoSuchFileException: /etc/logstash/queue/ApiUsage/checkpoint.1131", "backtrace"=>["sun/nio/fs/UnixException.java:86:in `translateToIOException'", "sun/nio/fs/UnixException.java:102:in `rethrowAsIOException'", "sun/nio/fs/UnixException.java:107:in `rethrowAsIOException'", "sun/nio/fs/UnixFileSystemProvider.java:214:in `newByteChannel'", "java/nio/file/Files.java:361:in `newByteChannel'", "java/nio/file/Files.java:407:in `newByteChannel'", "java/nio/file/Files.java:3152:in `readAllBytes'", "org/logstash/ackedqueue/io/FileCheckpointIO.java:56:in `read'", "org/logstash/ackedqueue/Queue.java:179:in `open'", "org/logstash/ackedqueue/ext/AbstractJRubyQueue.java:88:in `open'", "org/logstash/ackedqueue/ext/AbstractJRubyQueue$INVOKER$i$0$0$ruby_open.gen:-1:in `call'", "/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_acked_queue.rb:39:in `with_queue'", "/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_acked_queue.rb:28:in `create_file_based'", "/usr/share/logstash/logstash-core/lib/logstash/queue_factory.rb:29:in `create'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:172:in `initialize'", "org/jruby/RubyClass.java:1022:in `newInstance'", "org/jruby/RubyClass$INVOKER$i$newInstance.gen:-1:in `call'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in `block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in `block in converge_state'", "org/jruby/RubyArray.java:1734:in `each'", "org/jruby/RubyArray$INVOKER$i$0$0$each.gen:-1:in `call'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in `converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in `block in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in `converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:in `block in execute'", "org/jruby/RubyProc.java:289:in `call'", "org/jruby/RubyProc.java:273:in `call19'", "org/jruby/RubyProc$INVOKER$i$0$0$call19.gen:-1:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'", "org/jruby/RubyProc.java:289:in `call'", "org/jruby/RubyProc.java:246:in `call'", "java/lang/Thread.java:745:in `run'"]}

elastic cloud running 6.2.4
logstash version 6.2.4

any ideas?

2

What I'm now getting is mostly

[2018-05-01T13:41:31,424][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:ApiUsage, :exception=>"IOError", :message=>"org.logstash.LockException: The queue failed to obtain exclusive access, cause: Lock held by this virtual machine on lock path: /etc/logstash/queue/ApiUsage/.lock", :backtrace=>["org/logstash/ackedqueue/Queue.java:258:in `open'", "org/logstash/ackedqueue/ext/AbstractJRubyQueue.java:88:in `open'", "org/logstash/ackedqueue/ext/AbstractJRubyQueue$INVOKER$i$0$0$ruby_open.gen:-1:in `call'", "usr/share/logstash/logstash_minus_core/lib/logstash/util//usr/share/logstash/logstash-core/lib/logstash/util/wrapped_acked_queue.rb:39:in `invokeOther2:open'", "/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_acked_queue.rb:39:in `with_queue'", "usr/share/logstash/logstash_minus_core/lib/logstash/util//usr/share/logstash/logstash-core/lib/logstash/util/wrapped_acked_queue.rb:28:in `invokeOther2:with_queue'", "/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_acked_queue.rb:28:in `create_file_based'", "usr/share/logstash/logstash_minus_core/lib/logstash/util//usr/share/logstash/logstash-core/lib/logstash/util/wrapped_acked_queue.rb:-1:in `RUBY$method$create_file_based$0$__VARARGS__'", "usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/queue_factory.rb:29:in `invokeOther37:create_file_based'", "/usr/share/logstash/logstash-core/lib/logstash/queue_factory.rb:29:in `create'", "usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:172:in `invokeOther19:create'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:172:in `initialize'", "org/jruby/RubyClass.java:1015:in `newInstance'", "org/jruby/RubyClass$INVOKER$i$newInstance.gen:-1:in `call'", "usr/share/logstash/logstash_minus_core/lib/logstash/pipeline_action//usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `invokeOther9:new'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in `invokeOther5:execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in `block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in `invokeOther34:with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in `block in converge_state'", "org/jruby/RubyArray.java:1734:in `each'", "org/jruby/RubyArray$INVOKER$i$0$0$each.gen:-1:in `call'", "usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in `invokeOther56:each'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in `converge_state'", "usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in `invokeOther2:converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in `block in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in `invokeOther32:with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in `converge_state_and_update'", "usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/agent.rb:105:in `invokeOther1:converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:105:in `block in execute'", "org/jruby/RubyProc.java:289:in `call'", "org/jruby/RubyProc.java:273:in `call19'", "org/jruby/RubyProc$INVOKER$i$0$0$call19.gen:-1:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:18:in `interval'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:94:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:in `block in execute'", "org/jruby/RubyProc.java:289:in `call'", "org/jruby/RubyProc.java:273:in `call19'", "org/jruby/RubyProc$INVOKER$i$0$0$call19.gen:-1:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'", "org/jruby/RubyProc.java:289:in `call'", "org/jruby/RubyProc.java:246:in `call'", "java/lang/Thread.java:745:in `run'"]}

with the occasional

[2018-05-01T13:41:26,581][ERROR][logstash.inputs.metrics  ] Failed to create monitoring event {:message=>"For path: events. Map keys: [:pipelines, :reloads]", :error=>"LogStash::Instrument::MetricStore::MetricNotFound"}
3

not really sure what was wrong in it but stopped logstash, renamed the queue folder, restarted logstash and it's running correctly now. This means the 128Mb of data that was in those folders hasn't been logged but it is running again.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.