# No events in DLQ

**URL:** <https://discuss.elastic.co/t/no-events-in-dlq/94309>\
**Category:** Logstash\
**Created:** [July 24, 2017, 10:29am UTC](https://discuss.elastic.co/t/no-events-in-dlq/94309 "2017-07-24T10:29:48Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![AndreAga](https://avatars.discourse-cdn.com/v4/letter/a/ed8c4c/32.png) [@AndreAga](https://discuss.elastic.co/u/AndreAga)\
**Post date:** [July 24, 2017, 10:29am UTC](https://discuss.elastic.co/t/no-events-in-dlq/94309/1 "2017-07-24T10:29:48Z")

</div>

Hi all,  
I'm working with Logstash 5.5 and the new Feature DQL.

I've installed the plugin version **1.0.5** for LS 5.5:

/usr/share/logstash/bin/logstash-plugin list --verbose  
...  
logstash-input-dead\_letter\_queue (1.0.5)  
logstash-input-elasticsearch (4.0.4)  
...

This is the logstash.yml:

```
# ------------ Dead-Letter Queue Settings --------------
# Flag to turn on dead-letter queue.
#
dead_letter_queue.enable: true
#
# If using dead_letter_queue.enable: true, the directory path where the data files will be stored.
# Default is path.data/dead_letter_queue
#
path.dead_letter_queue: /data/logstashData/dead_letter_queue

```

In the dead\_letter\_queue dir, I have ten empty files (1.log ... 10.log) and in the same logstash pipeline I have 2 inputs:  
Kafka  
DQL

```
input {
  kafka {
    bootstrap_servers => "...:9092,...:9092"
    topics_pattern => "logstash-55*"  
    consumer_threads => 3
    decorate_events => true          
    codec => "json"
  }
  
  dead_letter_queue {
    path => "/data/logstashData/dead_letter_queue" 
    commit_offsets => true 
    codec => "json"
  }
}

```

In order to automatically handle dead events, because I could have some different mapping fields in JSON to ingest, I use a "log\_version" field that is incremented by a Ruby script in the pipeline that determines the name of the index (ex: logstash-\*-VERSION-YYYY.MM.DD).  
In this way, if an event could not be indexed, It's put into the DLQ with version=1, then it's read by DLQ input and the version is incremented by 1 (version=2) and so on.

Now, in Elasticsearch I have this exception: MapperParsingException: object mapping for [mapping\_request.payload] tried to parse field [payload] as object, but found a concrete value.

I think that this event must be in the DLQ but nothing is changed.

Logstash Pipeline API:

```
"inputs" : [ {
        "id" : "0fc849ea2666cb7428155be4777136ae64990ef7-2",
        "events" : {
          "out" : 0,
          "queue_push_duration_in_millis" : 0
        },
        "name" : "dead_letter_queue"
      }, {
        "id" : "0fc849ea2666cb7428155be4777136ae64990ef7-1",
        "events" : {
          "out" : 15209085,
          "queue_push_duration_in_millis" : 2000258
        },
        "name" : "kafka"
      } ],

```

How can I verify that DLQ is well-configured?

Thank you.

---

<div class="post-metadata">

**Author:** ![RobBavey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robbavey/32/20421_2.png) [@RobBavey](https://discuss.elastic.co/u/RobBavey)\
**Post date:** [July 25, 2017, 3:03pm UTC](https://discuss.elastic.co/t/no-events-in-dlq/94309/2 "2017-07-25T15:03:02Z")

</div>

Hi,

Couple of quick questions for you:

- Do you have an example of the data you are trying to ingest and the mapping (please remove any sensitive information)
- What version of the logstash-output-elasticsearch plugin are you using?
- Do you see any related error messages in the logstash log?
- Just to clarify - each of the .log files in the dead letter queue directory are empty, or a single byte?
- 
  - For your use cases, are you trying to mutate the fields to allow the entries to be indexed? (See [https://www.elastic.co/guide/en/logstash/current/dead-letter-queues.html#dlq-example](https://www.elastic.co/guide/en/logstash/current/dead-letter-queues.html#dlq-example) for an example)

Thanks!

Rob

---

<div class="post-metadata">

**Author:** ![AndreAga](https://avatars.discourse-cdn.com/v4/letter/a/ed8c4c/32.png) [@AndreAga](https://discuss.elastic.co/u/AndreAga)\
**Post date:** [July 25, 2017, 3:29pm UTC](https://discuss.elastic.co/t/no-events-in-dlq/94309/3 "2017-07-25T15:29:55Z")

</div>

Hi, thank you for your reply. Today it works. So now I'm able to process some (but not all) dead events.

About your questions

1. I'm cannot mutate the fields because I don't know which is the problematic field, I'm ingesting some json logs that are very very long and different. Is there a metadata that indicates the specific field? Otherwise I cannot process it and the easiest way is to create a separate index with the same pattern.
2. logstash-output-elasticsearch (7.3.6)
3. With the DLQ up&running sometimes I get this error:

> [2017-07-25T16:41:19,389][ERROR][logstash.pipeline] A plugin had an unrecoverable error. Will restart this plugin.  
> Plugin: \<LogStash::Inputs::DeadLetterQueue path=\>"/data/logstashData/dead\_letter\_queue", sincedb\_path=\>"/data/logstashDLQData/dlq\_offset", commit\_offsets=\>true, codec=\>\<LogStash::Codecs::JSON id=\>"json\_73337b45-c626-4cae-aab9-b2f4d8775a1e", enable\_metric=\>true, charset=\>"UTF-8"\>, id=\>"52ff26acd4e4d46066179db0a3cdf329cbe3c13c-1", enable\_metric=\>true, pipeline\_id=\>"main"\>  
> Error:  
> [2017-07-25T16:41:20,390][ERROR][logstash.pipeline] A plugin had an unrecoverable error. Will restart this plugin.  
> Plugin: \<LogStash::Inputs::DeadLetterQueue path=\>"/data/logstashData/dead\_letter\_queue", sincedb\_path=\>"/data/logstashDLQData/dlq\_offset", commit\_offsets=\>true, codec=\>\<LogStash::Codecs::JSON id=\>"json\_73337b45-c626-4cae-aab9-b2f4d8775a1e", enable\_metric=\>true, charset=\>"UTF-8"\>, id=\>"52ff26acd4e4d46066179db0a3cdf329cbe3c13c-1", enable\_metric=\>true, pipeline\_id=\>"main"\>  
> Error:  
> [2017-07-25T16:41:21,608][ERROR][logstash.pipeline] A plugin had an unrecoverable error. Will restart this plugin.  
> Plugin: \<LogStash::Inputs::DeadLetterQueue path=\>"/data/logstashData/dead\_letter\_queue", sincedb\_path=\>"/data/logstashDLQData/dlq\_offset", commit\_offsets=\>true, codec=\>\<LogStash::Codecs::JSON id=\>"json\_73337b45-c626-4cae-aab9-b2f4d8775a1e", enable\_metric=\>true, charset=\>"UTF-8"\>, id=\>"52ff26acd4e4d46066179db0a3cdf329cbe3c13c-1", enable\_metric=\>true, pipeline\_id=\>"main"\>  
> Error:  
> [2017-07-25T16:41:22,736][ERROR][logstash.pipeline] A plugin had an unrecoverable error. Will restart this plugin.  
> Plugin: \<LogStash::Inputs::DeadLetterQueue path=\>"/data/logstashData/dead\_letter\_queue", sincedb\_path=\>"/data/logstashDLQData/dlq\_offset", commit\_offsets=\>true, codec=\>\<LogStash::Codecs::JSON id=\>"json\_73337b45-c626-4cae-aab9-b2f4d8775a1e", enable\_metric=\>true, charset=\>"UTF-8"\>, id=\>"52ff26acd4e4d46066179db0a3cdf329cbe3c13c-1", enable\_metric=\>true, pipeline\_id=\>"main"\>  
> Error:  
> [2017-07-25T16:41:24,009][ERROR][logstash.pipeline] A plugin had an unrecoverable error. Will restart this plugin.  
> Plugin: \<LogStash::Inputs::DeadLetterQueue path=\>"/data/logstashData/dead\_letter\_queue", sincedb\_path=\>"/data/logstashDLQData/dlq\_offset", commit\_offsets=\>true, codec=\>\<LogStash::Codecs::JSON id=\>"json\_73337b45-c626-4cae-aab9-b2f4d8775a1e", enable\_metric=\>true, charset=\>"UTF-8"\>, id=\>"52ff26acd4e4d46066179db0a3cdf329cbe3c13c-1", enable\_metric=\>true, pipeline\_id=\>"main"\>  
> Error:  
> [2017-07-25T16:41:25,012][ERROR][logstash.pipeline] A plugin had an unrecoverable error. Will restart this plugin.  
> Plugin: \<LogStash::Inputs::DeadLetterQueue path=\>"/data/logstashData/dead\_letter\_queue", sincedb\_path=\>"/data/logstashDLQData/dlq\_offset", commit\_offsets=\>true, codec=\>\<LogStash::Codecs::JSON id=\>"json\_73337b45-c626-4cae-aab9-b2f4d8775a1e", enable\_metric=\>true, charset=\>"UTF-8"\>, id=\>"52ff26acd4e4d46066179db0a3cdf329cbe3c13c-1", enable\_metric=\>true, pipeline\_id=\>"main"\>  
> Error:  
> [2017-07-25T16:41:47,544][ERROR][logstash.pipeline] A plugin had an unrecoverable error. Will restart this plugin.  
> Plugin: \<LogStash::Inputs::DeadLetterQueue path=\>"/data/logstashData/dead\_letter\_queue", sincedb\_path=\>"/data/logstashDLQData/dlq\_offset", commit\_offsets=\>true, codec=\>\<LogStash::Codecs::JSON id=\>"json\_73337b45-c626-4cae-aab9-b2f4d8775a1e", enable\_metric=\>true, charset=\>"UTF-8"\>, id=\>"52ff26acd4e4d46066179db0a3cdf329cbe3c13c-1", enable\_metric=\>true, pipeline\_id=\>"main"\>  
> Error: invalid checksum of record

Please note that some json are very very long and some fields may contains the Base64 of PDF, etc. I really don't know the exact content.

Thank you.

---

<div class="post-metadata">

**Author:** ![RobBavey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robbavey/32/20421_2.png) [@RobBavey](https://discuss.elastic.co/u/RobBavey)\
**Post date:** [July 25, 2017, 5:29pm UTC](https://discuss.elastic.co/t/no-events-in-dlq/94309/4 "2017-07-25T17:29:25Z")

</div>

Hi,

Glad to hear you are having some success today! Thank you for the information - let's see if we can help you with some of your other issues

> [@AndreAga](#):
>
> I'm cannot mutate the fields because I don't know which is the problematic field, I'm ingesting some json logs that are very very long and different. Is there a metadata that indicates the specific field? Otherwise I cannot process it and the easiest way is to create a separate index with the same pattern.

The metadata includes the exception payload from Elasticsearch, but does not parse it into a separate metadata field - eg  
` "reason" => "Could not index event to Elasticsearch. status: 400, action: [\"index\", {:_id=>nil, :_index=>\"logstash-2017.06.22\", :_type=>\"logs\", :_routing=>nil}, 2017-06-22T01:29:29.804Z Suyogs-MacBook-Pro-2.local {\"geoip\":{\"location\":\"home\"}}], response: {\"index\"=>{\"_index\"=>\"logstash-2017.06.22\", \"_type\"=>\"logs\", \"_id\"=>\"AVzNayPze1iR9yDdI2MD\", \"status\"=>400, \"error\"=>{\"type\"=>\"mapper_parsing_exception\", \"reason\"=>\"failed to parse\", \"caused_by\"=>{\"type\"=>\"illegal_argument_exception\", \"reason\"=>\"illegal latitude value [266.30859375] for geoip.location\"}}}}"`

> [@AndreAga](#):
>
> [2017-07-25T16:41:19,389][ERROR][logstash.pipeline] A plugin had an unrecoverable error. Will restart this plugin.  
> Plugin: "/data/logstashData/dead\_letter\_queue", sincedb\_path=\>"/data/logstashDLQData/dlq\_offset", commit\_offsets=\>true, codec=\>"json\_73337b45-c626-4cae-aab9-b2f4d8775a1e", enable\_metric=\>true, charset=\>"UTF-8"\>, id=\>"52ff26acd4e4d46066179db0a3cdf329cbe3c13c-1", enable\_metric=\>true, pipeline\_id=\>"main"\>

This sounds like it could be a bug in the dead letter queue plugin. If you can reproduce this, would you mind running Logstash with debug logging (run with `--log.level=debug` on the command-line) and posting the same error, if it comes up, so we have more information to track down your issue.

Thanks,

Rob

---

<div class="post-metadata">

**Author:** ![AndreAga](https://avatars.discourse-cdn.com/v4/letter/a/ed8c4c/32.png) [@AndreAga](https://discuss.elastic.co/u/AndreAga)\
**Post date:** [July 26, 2017, 9:47am UTC](https://discuss.elastic.co/t/no-events-in-dlq/94309/5 "2017-07-26T09:47:23Z")

</div>

Hi,

Elasticsearch gave me this message 4/5 times today:

> MapperParsingException: object mapping for [mapping\_request.payload] tried to parse field [payload] as object, but found a concrete value

But logstash didn't put the events in the DLQ. Last update of DLQ's files is yesterday. No way to get them indexed 😕

If I use debug in logstash, It produces an amount of data that I cannot manage 😕

No other ways?

Thanks

---

<div class="post-metadata">

**Author:** ![RobBavey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robbavey/32/20421_2.png) [@RobBavey](https://discuss.elastic.co/u/RobBavey)\
**Post date:** [July 26, 2017, 2:50pm UTC](https://discuss.elastic.co/t/no-events-in-dlq/94309/6 "2017-07-26T14:50:43Z")

</div>

Hi,

I've filed a bug ticket for Logstash ([https://github.com/elastic/logstash/issues/7820](https://github.com/elastic/logstash/issues/7820)), because it looks like there could be an issue at our end. If you could review the bug entry and add as much information as you can to the ticket - log files, config files and any data that can help us reproduce the problem at our end would be hugely useful.

Thanks!

Roh

---

<div class="post-metadata">

**Author:** ![AndreAga](https://avatars.discourse-cdn.com/v4/letter/a/ed8c4c/32.png) [@AndreAga](https://discuss.elastic.co/u/AndreAga)\
**Post date:** [July 26, 2017, 2:52pm UTC](https://discuss.elastic.co/t/no-events-in-dlq/94309/7 "2017-07-26T14:52:19Z")

</div>

Thank you.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 23, 2017, 2:52pm UTC](https://discuss.elastic.co/t/no-events-in-dlq/94309/8 "2017-08-23T14:52:37Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.