# Why would DeadLetterQueue fail like this? **URL:** \ **Category:** Logstash\ **Created:** [April 11, 2018, 3:21pm UTC](https://discuss.elastic.co/t/why-would-deadletterqueue-fail-like-this/127664 "2018-04-11T15:21:07Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![smbullet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/smbullet/32/27833_2.png) [@smbullet](https://discuss.elastic.co/u/smbullet)\ **Post date:** [April 11, 2018, 3:21pm UTC](https://discuss.elastic.co/t/why-would-deadletterqueue-fail-like-this/127664/1 "2018-04-11T15:21:07Z")
``` [2018-04-11T10:37:31,479][ERROR][logstash.pipeline] A plugin had an unrecoverable error. Will restart this plugin. Pipeline_id:main Plugin: "/home/logstash_user/logstash/data/dead_letter_queue", add_field=>{"original"=>"true"}, id=>"577895150ca70c065d4a2106f80aa61b5b7dea788959e2c3f2ee4dd4fe41713f", enable_metric=>true, codec=>"plain_449ee892-6c93-499e-9622-9aca0cc38d8d", enable_metric=>true, charset=>"UTF-8">, pipeline_id=>"main", commit_offsets=>true> Error: Exception: Java::JavaLang::ArrayIndexOutOfBoundsException Stack: java.util.zip.CRC32.update(java/util/zip/CRC32.java:74) org.logstash.common.io.RecordIOReader.getRecord(org/logstash/common/io/RecordIOReader.java:214) org.logstash.common.io.RecordIOReader.readEvent(org/logstash/common/io/RecordIOReader.java:233) org.logstash.common.io.DeadLetterQueueReader.pollEntryBytes(org/logstash/common/io/DeadLetterQueueReader.java:120) org.logstash.common.io.DeadLetterQueueReader.pollEntry(org/logstash/common/io/DeadLetterQueueReader.java:97) org.logstash.input.DeadLetterQueueInputPlugin.run(org/logstash/input/DeadLetterQueueInputPlugin.java:82) java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498) org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:423) org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:290) RUBY.run(/home/logstash_user/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-dead_letter_queue-1.1.4/lib/logstash/inputs/dead_letter_queue.rb:60) RUBY.inputworker(/home/logstash_user/logstash/logstash-core/lib/logstash/pipeline.rb:516) RUBY.block in start_input(/home/logstash_user/logstash/logstash-core/lib/logstash/pipeline.rb:509) org.jruby.RubyProc.call(org/jruby/RubyProc.java:289) org.jruby.RubyProc.call(org/jruby/RubyProc.java:246) java.lang.Thread.run(java/lang/Thread.java:748) ``` This is failure seems to occur here: [https://github.com/elastic/logstash/blob/master/logstash-core/src/main/java/org/logstash/common/io/RecordIOReader.java#L214](https://github.com/elastic/logstash/blob/master/logstash-core/src/main/java/org/logstash/common/io/RecordIOReader.java#L214) I also get this: ``` [2018-04-11T10:37:32,483][ERROR][logstash.pipeline] A plugin had an unrecoverable error. Will restart this plugin. Pipeline_id:main Plugin: "/home/logstash_user/logstash/data/dead_letter_queue", add_field=>{"original"=>"true"}, id=>"577895150ca70c065d4a2106f80aa61b5b7dea788959e2c3f2ee4dd4fe41713f", enable_metric=>true, codec=>"plain_449ee892-6c93-499e-9622-9aca0cc38d8d", enable_metric=>true, charset=>"UTF-8">, pipeline_id=>"main", commit_offsets=>true> Error: Exception: Java::JavaLang::IllegalArgumentException Stack: java.nio.Buffer.position(java/nio/Buffer.java:244) org.logstash.common.io.RecordIOReader.seekToStartOfEventInBlock(org/logstash/common/io/RecordIOReader.java:173) org.logstash.common.io.RecordIOReader.consumeToStartOfEvent(org/logstash/common/io/RecordIOReader.java:192) org.logstash.common.io.RecordIOReader.readEvent(org/logstash/common/io/RecordIOReader.java:226) org.logstash.common.io.DeadLetterQueueReader.pollEntryBytes(org/logstash/common/io/DeadLetterQueueReader.java:120) org.logstash.common.io.DeadLetterQueueReader.pollEntry(org/logstash/common/io/DeadLetterQueueReader.java:97) org.logstash.input.DeadLetterQueueInputPlugin.run(org/logstash/input/DeadLetterQueueInputPlugin.java:82) java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498) org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:423) org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:290) RUBY.run(/home/logstash_user/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-dead_letter_queue-1.1.4/lib/logstash/inputs/dead_letter_queue.rb:60) RUBY.inputworker(/home/logstash_user/logstash/logstash-core/lib/logstash/pipeline.rb:516) RUBY.block in start_input(/home/logstash_user/logstash/logstash-core/lib/logstash/pipeline.rb:509) org.jruby.RubyProc.call(org/jruby/RubyProc.java:289) org.jruby.RubyProc.call(org/jruby/RubyProc.java:246) java.lang.Thread.run(java/lang/Thread.java:748) ``` Which seems to fail here: [https://github.com/elastic/logstash/blob/master/logstash-core/src/main/java/org/logstash/common/io/RecordIOReader.java#L173](https://github.com/elastic/logstash/blob/master/logstash-core/src/main/java/org/logstash/common/io/RecordIOReader.java#L173) And this: ``` [2018-04-11T10:42:12,976][ERROR][logstash.pipeline] A plugin had an unrecoverable error. Will restart this plugin. Pipeline_id:main Plugin: "/home/logstash_user/logstash/data/dead_letter_queue", add_field=>{"original"=>"true"}, id=>"577895150ca70c065d4a2106f80aa61b5b7dea788959e2c3f2ee4dd4fe41713f", enable_metric=>true, codec=>"plain_449ee892-6c93-499e-9622-9aca0cc38d8d", enable_metric=>true, charset=>"UTF-8">, pipeline_id=>"main", commit_offsets=>true> Error: invalid checksum of record Exception: Java::JavaLang::RuntimeException Stack: org.logstash.common.io.RecordIOReader.getRecord(org/logstash/common/io/RecordIOReader.java:217) org.logstash.common.io.RecordIOReader.readEvent(org/logstash/common/io/RecordIOReader.java:233) org.logstash.common.io.DeadLetterQueueReader.pollEntryBytes(org/logstash/common/io/DeadLetterQueueReader.java:120) org.logstash.common.io.DeadLetterQueueReader.pollEntry(org/logstash/common/io/DeadLetterQueueReader.java:97) org.logstash.input.DeadLetterQueueInputPlugin.run(org/logstash/input/DeadLetterQueueInputPlugin.java:82) java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498) org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:423) org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:290) RUBY.run(/home/logstash_user/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-dead_letter_queue-1.1.4/lib/logstash/inputs/dead_letter_queue.rb:60) RUBY.inputworker(/home/logstash_user/logstash/logstash-core/lib/logstash/pipeline.rb:516) RUBY.block in start_input(/home/logstash_user/logstash/logstash-core/lib/logstash/pipeline.rb:509) org.jruby.RubyProc.call(org/jruby/RubyProc.java:289) org.jruby.RubyProc.call(org/jruby/RubyProc.java:246) java.lang.Thread.run(java/lang/Thread.java:748) ``` Which throws here: [https://github.com/elastic/logstash/blob/master/logstash-core/src/main/java/org/logstash/common/io/RecordIOReader.java#L217](https://github.com/elastic/logstash/blob/master/logstash-core/src/main/java/org/logstash/common/io/RecordIOReader.java#L217) Any suggestions on how to debug this? It's difficult for me to try to intercept the message because it all gets thrown in Java, not Jruby. ---
**Author:** ![RobBavey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robbavey/32/20421_2.png) [@RobBavey](https://discuss.elastic.co/u/RobBavey)\ **Post date:** [April 13, 2018, 6:36pm UTC](https://discuss.elastic.co/t/why-would-deadletterqueue-fail-like-this/127664/2 "2018-04-13T18:36:39Z")
Hi @smbullet, Can you share your logstash configuration files (with any sensitive information removed) and which version you are running? Thanks, Rob ---
**Author:** ![smbullet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/smbullet/32/27833_2.png) [@smbullet](https://discuss.elastic.co/u/smbullet)\ **Post date:** [April 16, 2018, 3:13pm UTC](https://discuss.elastic.co/t/why-would-deadletterqueue-fail-like-this/127664/3 "2018-04-16T15:13:09Z")
Hi Rob, Thanks for the reply! It seems that I had a bunch of messages piling up in the DLQ due to poorly formatted data I was sending to Elasticsearch. Once I fixed that, everything started working perfectly and I can't seem to repeat the issue. I'll update this if I figure out more. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 14, 2018, 3:13pm UTC](https://discuss.elastic.co/t/why-would-deadletterqueue-fail-like-this/127664/4 "2018-05-14T15:13:15Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.