# Logstash DLQ dir empty, but all events end up duplicated in the Elasticsearch's dlq index

**URL:** <https://discuss.elastic.co/t/logstash-dlq-dir-empty-but-all-events-end-up-duplicated-in-the-elasticsearchs-dlq-index/312250>\
**Category:** Logstash\
**Created:** [August 17, 2022, 6:22am UTC](https://discuss.elastic.co/t/logstash-dlq-dir-empty-but-all-events-end-up-duplicated-in-the-elasticsearchs-dlq-index/312250 "2022-08-17T06:22:41Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![kudlatyjoe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kudlatyjoe/32/109759_2.png) [@kudlatyjoe](https://discuss.elastic.co/u/kudlatyjoe)\
**Post date:** [August 17, 2022, 6:22am UTC](https://discuss.elastic.co/t/logstash-dlq-dir-empty-but-all-events-end-up-duplicated-in-the-elasticsearchs-dlq-index/312250/1 "2022-08-17T06:22:41Z")

</div>

Hi all,

I'm currently trying to introduce the dead-letter-plugin to the ELK stack that we use for collecting application logs.  
We've been having some issues with the logging functionality breaking down and losing some requests, hence the idea for adding a DLQ to the pipeline to at least have a peek into what may be causing us troubles.

Here's the basic setup that I've got running locally. The whole stack is dockerized.  
_logstash.conf_

```auto
input {
  beats {
    port => 5044
  }
}

filter {
  json {
    source => "message"
  }
  date{
    match => ["timestamp", "UNIX_MS"]
    target => "@timestamp"
  }
  ruby {
    code => "event.set('indexDay', event.get('[@timestamp]').time.localtime('+09:00').strftime('%Y%m%d'))"
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    template => "/usr/share/logstash/templates/logstash.template.json"
    template_name => "logstash"
    template_overwrite => true
    index => "logstash-%{indexDay}"
    codec => json
  }
  stdout {
    codec => rubydebug
  }
}

```

_logstash\_dlq.conf_

```auto
input {
  dead_letter_queue {
    path => "/usr/share/logstash/data/dead_letter_queue"
    commit_offsets => true 
    pipeline_id => "main" 
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "logstash-dlq-%{indexDay}"
    codec => json
  }
}

```

_logstash.yml_

```auto
dead_letter_queue:
  enable: true

```

The whole thing seems to be working, but not really how I'd expect it. All the logs that are being processed by logstash end up in both indices e.g. logstash-20220817 and logstash-dlq-20220817. Also the logstash's directory for keeping the logs _data/dead\_letter\_queue/main_ has only one entry available - 1.log and its whole content is '1'.

I'd appreciate any tips that might help me set this up properly

Cheers,  
Adam

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 14, 2022, 6:23am UTC](https://discuss.elastic.co/t/logstash-dlq-dir-empty-but-all-events-end-up-duplicated-in-the-elasticsearchs-dlq-index/312250/2 "2022-09-14T06:23:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.