# Logstash DLQ: sincedb never gets created

**URL:** <https://discuss.elastic.co/t/logstash-dlq-sincedb-never-gets-created/370205>\
**Category:** Logstash\
**Created:** [November 8, 2024, 8:13am UTC](https://discuss.elastic.co/t/logstash-dlq-sincedb-never-gets-created/370205 "2024-11-08T08:13:15Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![hilsonp](https://avatars.discourse-cdn.com/v4/letter/h/f17d59/32.png) [@hilsonp](https://discuss.elastic.co/u/hilsonp)\
**Post date:** [November 8, 2024, 8:13am UTC](https://discuss.elastic.co/t/logstash-dlq-sincedb-never-gets-created/370205/1 "2024-11-08T08:13:15Z")

</div>

Hello,

I'm trying to configure dead letter queue on a setup which was installed by another team a few years ago.

Documents refused by elastic are well going to the dlq.  
The dlq pipeline ingests them and send them correctly formatted to the index created for receiving the dlq events for analysis.

My problème is that at each restart of logstash, the events are re-sents to elastic although I set both commit\_offsets and clean\_consumed to true.  
The sincedb file never appear whatever combination of settings I do.

I highly suspect that this setup was first used running logstash as root and then logstash as logstash user.  
My suspiscion comes from the fact that running the logstash as root to test my pipeline created the sincedb.

We are running LogStash 8.4.2.

Our settings are the following:

/etc/logstash/logstash.yml

```auto
path.data:
  /var/lib/logstash
#pipeline.id: filestream-apigee
pipeline.ordered: auto
pipeline.ecs_compatibility: v8
config.reload.automatic: false
config.reload.interval: 60s
config.debug: true
log.level: debug
path.logs: /home/elasticsearch/log/logstash
xpack.management.elasticsearch.ssl.certificate_authority: "/etc/elasticsearch/certs/ca/ca.crt"
dead_letter_queue.enable: true
path.dead_letter_queue: "/home/sites/logstash/data/dead_letter_queue"

```

/etc/logstash/pipelines.yml

```auto
- pipeline.id: "filestream-apigee"
  path.config: "/etc/logstash/conf.d/filestream-apigee.conf"

- pipeline.id: "dead-letter-queue"
  path.config: "/etc/logstash/conf.d/dead-letter-queue.conf"

```

/etc/logstash/conf.d/dead-letter-queue.conf

```auto
input {
  dead_letter_queue {
    path => "/home/sites/logstash/data/dead_letter_queue/"
    pipeline_id => "filestream-apigee"
    commit_offsets => true
    clean_consumed => true
    sincedb_path => "/home/sites/logstash/data/dead_letter_queue/filestream-apigee/.sincedb_manually_configured"
  }
}
filter {
  prune {
    add_field => {
      "[_dlq][entry_time]" => "%{[@metadata][dead_letter_queue][entry_time]}"
      "[_dlq][plugin_id]" => "%{[@metadata][dead_letter_queue][plugin_id]}"
      "[_dlq][plugin_type]" => "%{[@metadata][dead_letter_queue][plugin_type]}"
      "[_dlq][reason]" => "%{[@metadata][dead_letter_queue][reason]}"
    }
    whitelist_names => ["@timestamp", "dlq", "startTime", "endTime", "issuer", "tags", "input", "correlationId", "log", "fields", "host", "agent"]
  }
  date {
    match => ["[_dlq][entry_time]" , "ISO8601", "yyyy-MM-dd'T'HH:mm:ss.SSS" ]
    target => "@timestamp"
    timezone => "Europe/Brussels"
  }
}
output {
  elasticsearch {
    hosts => ["https://192.168.1.2:9200"]
    index => "dead-letter-queue"
    cacert => "/etc/elasticsearch/certs/ca/ca.crt"
    action => "create"
    user => "logstash_writer"
    password => "logstash_writer_pwd"
    ssl => true
  }
  stdout {
     codec => rubydebug
  }
}

```

Here are the rights on the dlq folder (within which the sincedb should be created).

```auto
dr-xr-xr-x. 19 root root 4096 Jun 10 15:44 /
drwxr-xr-x. 25 root root 4096 Jan 16 2024 /home
drwxr-x--- 6 root sites 4096 Mar 25 2024 /home/sites
drwxrwx--- 3 logstash logstash 4096 Sep 24 15:17 /home/sites/logstash
drwxr-x--- 3 logstash logstash 4096 Sep 24 15:17 /home/sites/logstash/data
drwxr-x--- 4 logstash logstash 4096 Nov 7 23:37 /home/sites/logstash/data/dead_letter_queue
drwxr-xr-x 2 logstash logstash 4096 Nov 7 23:52 /home/sites/logstash/data/dead_letter_queue/filestream-apigee

```

I did try to set the commit\_offsets to false with clean\_consumed to true. As expected, I got a configuration error saying that both must be true which indicates me that the version I run know about those settings.

I tried to find my way through the log but did not find any warning or error.

What else can I check ?

Thank you in advance.