I have a
main pipeline that will sometimes send docs to its Dead Letter Queue. I have another
fixes pipeline that fixes those issues. However, even though I can see that
fixes is fixing and successfully processing the docs, the docs remain in the
main Dead Letter Queue directory.
Because of this, Logstash reaches the maxQueueSize of the directory and stops processing them. Shouldn't Logstash be removing the ones that are successfully indexed so that the DLQ doesn't keep increasing and increasing until it hits that limit?