Why would DeadLetterQueue fail like this?

smbullet · April 11, 2018, 3:21pm

[2018-04-11T10:37:31,479][ERROR][logstash.pipeline        ] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::DeadLetterQueue path=>"/home/logstash_user/logstash/data/dead_letter_queue", add_field=>{"original"=>"true"}, id=>"577895150ca70c065d4a2106f80aa61b5b7dea788959e2c3f2ee4dd4fe41713f", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_449ee892-6c93-499e-9622-9aca0cc38d8d", enable_metric=>true, charset=>"UTF-8">, pipeline_id=>"main", commit_offsets=>true>
  Error:
  Exception: Java::JavaLang::ArrayIndexOutOfBoundsException
  Stack: java.util.zip.CRC32.update(java/util/zip/CRC32.java:74)
org.logstash.common.io.RecordIOReader.getRecord(org/logstash/common/io/RecordIOReader.java:214)
org.logstash.common.io.RecordIOReader.readEvent(org/logstash/common/io/RecordIOReader.java:233)
org.logstash.common.io.DeadLetterQueueReader.pollEntryBytes(org/logstash/common/io/DeadLetterQueueReader.java:120)
org.logstash.common.io.DeadLetterQueueReader.pollEntry(org/logstash/common/io/DeadLetterQueueReader.java:97)
org.logstash.input.DeadLetterQueueInputPlugin.run(org/logstash/input/DeadLetterQueueInputPlugin.java:82)
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:423)
org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:290)
RUBY.run(/home/logstash_user/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-dead_letter_queue-1.1.4/lib/logstash/inputs/dead_letter_queue.rb:60)
RUBY.inputworker(/home/logstash_user/logstash/logstash-core/lib/logstash/pipeline.rb:516)
RUBY.block in start_input(/home/logstash_user/logstash/logstash-core/lib/logstash/pipeline.rb:509)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:246)
java.lang.Thread.run(java/lang/Thread.java:748)

This is failure seems to occur here: https://github.com/elastic/logstash/blob/master/logstash-core/src/main/java/org/logstash/common/io/RecordIOReader.java#L214

I also get this:

[2018-04-11T10:37:32,483][ERROR][logstash.pipeline        ] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::DeadLetterQueue path=>"/home/logstash_user/logstash/data/dead_letter_queue", add_field=>{"original"=>"true"}, id=>"577895150ca70c065d4a2106f80aa61b5b7dea788959e2c3f2ee4dd4fe41713f", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_449ee892-6c93-499e-9622-9aca0cc38d8d", enable_metric=>true, charset=>"UTF-8">, pipeline_id=>"main", commit_offsets=>true>
  Error:
  Exception: Java::JavaLang::IllegalArgumentException
  Stack: java.nio.Buffer.position(java/nio/Buffer.java:244)
org.logstash.common.io.RecordIOReader.seekToStartOfEventInBlock(org/logstash/common/io/RecordIOReader.java:173)
org.logstash.common.io.RecordIOReader.consumeToStartOfEvent(org/logstash/common/io/RecordIOReader.java:192)
org.logstash.common.io.RecordIOReader.readEvent(org/logstash/common/io/RecordIOReader.java:226)
org.logstash.common.io.DeadLetterQueueReader.pollEntryBytes(org/logstash/common/io/DeadLetterQueueReader.java:120)
org.logstash.common.io.DeadLetterQueueReader.pollEntry(org/logstash/common/io/DeadLetterQueueReader.java:97)
org.logstash.input.DeadLetterQueueInputPlugin.run(org/logstash/input/DeadLetterQueueInputPlugin.java:82)
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:423)
org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:290)
RUBY.run(/home/logstash_user/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-dead_letter_queue-1.1.4/lib/logstash/inputs/dead_letter_queue.rb:60)
RUBY.inputworker(/home/logstash_user/logstash/logstash-core/lib/logstash/pipeline.rb:516)
RUBY.block in start_input(/home/logstash_user/logstash/logstash-core/lib/logstash/pipeline.rb:509)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:246)
java.lang.Thread.run(java/lang/Thread.java:748)

Which seems to fail here: https://github.com/elastic/logstash/blob/master/logstash-core/src/main/java/org/logstash/common/io/RecordIOReader.java#L173

And this:

[2018-04-11T10:42:12,976][ERROR][logstash.pipeline        ] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::DeadLetterQueue path=>"/home/logstash_user/logstash/data/dead_letter_queue", add_field=>{"original"=>"true"}, id=>"577895150ca70c065d4a2106f80aa61b5b7dea788959e2c3f2ee4dd4fe41713f", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_449ee892-6c93-499e-9622-9aca0cc38d8d", enable_metric=>true, charset=>"UTF-8">, pipeline_id=>"main", commit_offsets=>true>
  Error: invalid checksum of record
  Exception: Java::JavaLang::RuntimeException
  Stack: org.logstash.common.io.RecordIOReader.getRecord(org/logstash/common/io/RecordIOReader.java:217)
org.logstash.common.io.RecordIOReader.readEvent(org/logstash/common/io/RecordIOReader.java:233)
org.logstash.common.io.DeadLetterQueueReader.pollEntryBytes(org/logstash/common/io/DeadLetterQueueReader.java:120)
org.logstash.common.io.DeadLetterQueueReader.pollEntry(org/logstash/common/io/DeadLetterQueueReader.java:97)
org.logstash.input.DeadLetterQueueInputPlugin.run(org/logstash/input/DeadLetterQueueInputPlugin.java:82)
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:423)
org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:290)
RUBY.run(/home/logstash_user/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-dead_letter_queue-1.1.4/lib/logstash/inputs/dead_letter_queue.rb:60)
RUBY.inputworker(/home/logstash_user/logstash/logstash-core/lib/logstash/pipeline.rb:516)
RUBY.block in start_input(/home/logstash_user/logstash/logstash-core/lib/logstash/pipeline.rb:509)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:246)
java.lang.Thread.run(java/lang/Thread.java:748)

Which throws here: https://github.com/elastic/logstash/blob/master/logstash-core/src/main/java/org/logstash/common/io/RecordIOReader.java#L217

Any suggestions on how to debug this? It's difficult for me to try to intercept the message because it all gets thrown in Java, not Jruby.

RobBavey · April 13, 2018, 6:36pm

Hi @smbullet,

Can you share your logstash configuration files (with any sensitive information removed) and which version you are running?

Thanks,

Rob

smbullet · April 16, 2018, 3:13pm

Hi Rob,

Thanks for the reply! It seems that I had a bunch of messages piling up in the DLQ due to poorly formatted data I was sending to Elasticsearch. Once I fixed that, everything started working perfectly and I can't seem to repeat the issue. I'll update this if I figure out more.

system · May 14, 2018, 3:13pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
DeadLetterQueue Plugin Error Logstash	1	451	May 2, 2019
Best way to debug DeadLetterQueue crashing Logstash	1	317	April 23, 2018
Logstash Error - Deadletter queue/plugin - troubleshooting Logstash	2	716	January 30, 2019
Is there any other way to store rejected records by elasticsearch? Logstash	1	339	December 7, 2018
Logstash deadletterqueue Logstash	4	280	February 27, 2019

Why would DeadLetterQueue fail like this?

Related topics