Why does Logstash not clear DLQ after successful fixes?

Elastic Stack Logstash
1

I have a main pipeline that will sometimes send docs to its Dead Letter Queue. I have another fixes pipeline that fixes those issues. However, even though I can see that fixes is fixing and successfully processing the docs, the docs remain in the main Dead Letter Queue directory.

Because of this, Logstash reaches the maxQueueSize of the directory and stops processing them. Shouldn't Logstash be removing the ones that are successfully indexed so that the DLQ doesn't keep increasing and increasing until it hits that limit?

2

Unfortunately you'll need to stop logstash and remove it manually today. We're working on making the DLQ automatically rotate. Please track: https://github.com/elastic/logstash/issues/6990

I agree that we should also add the feature you've requested, letting the DLQ input remove processed messages.

3

I've created this issue here to track: https://github.com/elastic/logstash/issues/8904

2 Likes
4

Thanks! Is it necessary to stop Logstash as a service first? Or can I just rm dead_letter_queue/*?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.