I just started working with the Dead Letter Queue, but after restarting my pipelines a few times I am now getting this message repeatedly in my logs:
[WARN ][org.logstash.common.io.DeadLetterQueueWriter] Event previously submitted to dead letter queue. Skipping...
I tried deleting all the files in the dead_letter_queue directories, but that doesn't seem to get it to start from the beginning again. Any help would be appreciated.
@warkolm any thoughts on this? sorry to mention you, i just really need help with this one and i feel like i've tried everything and read all the docs.
Edit: Looks like the issue was that I was actually clearing the queue, but the warning was because the DLQ pipeline was not resolving the mapping issues for the documents. When I read the warning log message I thought it was coming from the DLQ pipeline the first timeit tried to process the event.
It would be awesome if it were more verbose. All I saw were Dead Letter Queue warnings. I didn't see any of the usual "Elasticsearch couldn't index..." warnings. I'm not sure why those weren't included. Does DLQ being turned on suppress those warnings or something?