Whenever the logstash is restarted, getting plugin error as below
23:46:27.262 [[main]<dead_letter_queue] ERROR logstash.pipeline - A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::DeadLetterQueue path=>"/var/log/logstash/deadletter", commit_offsets=>true, pipeline_id=>"-------", id=>"--------", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_56afa592-a052-4899-b8d7-32ba2a094fbc", enable_metric=>true, charset=>"UTF-8">>
Error: Invalid file. check version
Exception: Java::JavaLang::RuntimeException
Stack: org.logstash.common.io.RecordIOReader.(org/logstash/common/io/RecordIOReader.java:59)
org.logstash.common.io.DeadLetterQueueReader.pollEntryBytes(org/logstash/common/io/DeadLetterQueueReader.java:126)
org.logstash.common.io.DeadLetterQueueReader.pollEntry(org/logstash/common/io/DeadLetterQueueReader.java:97)
org.logstash.input.DeadLetterQueueInputPlugin.run(org/logstash/input/DeadLetterQueueInputPlugin.java:82)
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:423)
org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:290)
RUBY.run(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-dead_letter_queue-1.1.4/lib/logstash/inputs/dead_letter_queue.rb:60)
usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.invokeOther8:run(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:514)
usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.inputworker(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:514)
usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.invokeOther0:inputworker(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:507)
usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.block in start_input(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:507)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:246)
java.lang.Thread.run(java/lang/Thread.java:748)
23:46:27.422 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] INFO logstash.pipeline - Starting pipeline {:pipeline_id=>"--------", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
Tried to enable DEBUG logs for "org.logstash.common.io.DeadLetterQueueReader", but not debug entries are found. Not sure if my log4J settings are correct. Also, not able to identify the cause of issue.
cat log4j2.properties
logger.DeadLetterQueueReader.name = org.logstash.common.io.DeadLetterQueueReader
logger.DeadLetterQueueReader.level = debug
I have detailed various scenarios on the same issue in the thread Logstash Error - Deadletter queue/plugin - troubleshooting but no response
Any guidance around the issue would be helpful.