DLQ rotate or retention

[Aviad_Bitton]

Hi,
I am using Logstash 6.8 and i intend to use dlq for this purpose:
If an event is not entered the Elastic cluster because of mapping conflict or something else,
i want to show that log in a specific index that tells why, I am able to get the reason and show it in Kibana. So this is great.

What i am struggling about is the retention of those dlq logs in the logstash server. It does not clear itself, and only got a limit on the size of it.

Is there any option to do retention or rotate to those logs so that the dlq can accept events and not limit it and ignore them?

Thank you.

[Badger]

I have not used a DLQ, but my understanding is that the disk space consumed by the queue does not get freed. But you have set the maximum size which implies you are OK with it consuming that space.

If it fails to index a document (with some errors) the elasticsearch output writes the event to a DLQ. You must then run some other pipeline that consumes events using a dead_letter_queue input. If you set the commit_offsets option to true then the disk space used by the events you are consuming is made available for the elasticsearch output to write to. It is not freed on the file system. The file still exists, it just no longer contains events.

If you no longer expect to get additional events written to the DLQ (perhaps because you have fixed the mappings in your index templates) I would expect the code to be robust enough that if you stopped logstash and removed all the DLQ files it would come back up without a problem. You should definitely try that on a lab system first though