Dear All,
Trying to trouble shoot the logstash issues that I'm facing, and any hints would be helpful
Background
Note the issue is in both prod and dev, and the numbers below are from dev
Log ES Version : 6.2.1
Monitoring ES Version : 6.3.0
Logstash version: docker.elastic.co:443/logstash/logstash 6.2.4
Both logging elastic, and logstash send monitoring data to Monitoring ES.
Problems
-
During logstash setup, we get the below error
[ERROR] 2018-12-10 19:50:52.427 [[main]-pipeline-manager] pipeline - Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::Inputs::DeadLetterQueue path=>"/var/log/logstash/deadletter", commit_offsets=>true, pipeline_id=>"MYPIPELINE", id=>"a429131198fb77d9085a00b204b1d5de364499af7ede8a7f5939a685021acb58", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_1fe8d87a-f2de-45d8-adee-e3b2dc99023b", enable_metric=>true, charset=>"UTF-8">>", :error=>"/var/log/logstash/deadletter/MYPIPELINE", :thread=>"#<Thread:0x6eb4570e@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}
[ERROR] 2018-12-10 19:50:53.390 [[main]-pipeline-manager] pipeline - Pipeline aborted due to error {:pipeline_id=>"main", :exception=>java.nio.file.NoSuchFileException: /var/log/logstash/deadletter/MYPIPELINE, :backtrace=>["sun.nio.fs.UnixException.translateToIOException(sun/nio/fs/UnixException.java:86)", "sun.nio.fs.UnixException.asIOException(sun/nio/fs/UnixException.java:111)", "sun.nio.fs.LinuxWatchService$Poller.implRegister(sun/nio/fs/LinuxWatchService.java:246)", "sun.nio.fs.AbstractPoller.processRequests(sun/nio/fs/AbstractPoller.java:260)", "sun.nio.fs.LinuxWatchService$Poller.run(sun/nio/fs/LinuxWatchService.java:364)", "java.lang.Thread.run(java/lang/Thread.java:748)"], :thread=>"#<Thread:0x6eb4570e@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}
[ERROR] 2018-12-10 19:50:53.394 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: LogStash::PipelineAction::Create/pipeline_id:main, action_result: false", :backtrace=>nil}
The logstash is in multi-pipeline mode, and we have the deadletter queues configured using a separate pipeline "deadletter.conf" with appropriate inputs/outputs representing all other pipelines. The deadletter pipeline, which runs as "main" sends the deadletter data to ES Log against a separate indice with no strict mapping. The error indicates that one of the pipeline is missing. Is looks like it is due to the fact that deadletter pipeline is also started along with other pipeline initially. This comes only during logstash provisioning and start after deploying all pipelines. Not sure if the observation is correct.
- Restart Logstash (docker instance), and below error is observed during restart. Not sure why this error is coming for each pipeline that is getting recoginized. Finally all pipelines including "main" that represent deadletter is running.
[ERROR] 2018-12-10 21:26:19.002 [[main]<dead_letter_queue] pipeline - A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::DeadLetterQueue path=>"/var/log/logstash/deadletter", commit_offsets=>true, pipeline_id=>"FOR EACH PIPELINE", id=>"8642a2efb7b01a2d283c862ad0d50b7eaee3e354425bd3026467d66af5546790", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_4866de30-3254-43fb-9a9a-0ea75ec572a1", enable_metric=>true, charset=>"UTF-8">>
Error: Invalid file. check version
Exception: Java::JavaLang::RuntimeException
Stack: org.logstash.common.io.RecordIOReader.(org/logstash/common/io/RecordIOReader.java:59)
org.logstash.common.io.DeadLetterQueueReader.pollEntryBytes(org/logstash/common/io/DeadLetterQueueReader.java:126)
org.logstash.common.io.DeadLetterQueueReader.pollEntry(org/logstash/common/io/DeadLetterQueueReader.java:97)
org.logstash.input.DeadLetterQueueInputPlugin.run(org/logstash/input/DeadLetterQueueInputPlugin.java:82)
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:423)
org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:290)
RUBY.run(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-dead_letter_queue-1.1.4/lib/logstash/inputs/dead_letter_queue.rb:60)
usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.invokeOther8:run(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:514)
usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.inputworker(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:514)
usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.invokeOther0:inputworker(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:507)
usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.block in start_input(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:507)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:246)
java.lang.Thread.run(java/lang/Thread.java:748)
Any help in identifying the cause and recover from these errors would be helpful
