Dead Letter Queue (DLQ) issue

Elastic Stack Logstash
1

Hello,

I'm using Logstash 6.3.2 and I've enabled the DLQ but it doesn't appear to be indexing the events back into Elasticsearch; the log files in the DLQ data directory just keep growing and all i'm seeing in logstash-plain.log is:

[WARN ][org.logstash.common.io.DeadLetterQueueWriter] Event previously submitted to dead letter queue. Skipping...

I have the following in my logstash.conf file to repair the DLQ messages so they can be indexed into Elasticsearch:

if "dlq" in [tags] {
      mutate {
        convert => {
          "src_ip" => "string"
          "dst_ip" => "string"
       }          
     }
    }

However the repaired messages I'm expecting to see in Elasticsearch aren't there.

Can anyone help?

2

If I remove the problematic fields as below:

if "dlq" in [tags] {
     mutate { remove_field => [ "src_ip", "dst_ip" ] } 
     }

I no longer see the "Event previously submitted to dead letter queue. Skipping..." error in the Logstash log, however the message is still not indexed into Elasticsearch.

Anyone?