[WARN ][org.logstash.common.io.DeadLetterQueueWriter] Event previously submitted to dead letter queue. Skipping


I just started working with the Dead Letter Queue, but after restarting my pipelines a few times I am now getting this message repeatedly in my logs:

[WARN ][org.logstash.common.io.DeadLetterQueueWriter] Event previously submitted to dead letter queue. Skipping...

I tried deleting all the files in the dead_letter_queue directories, but that doesn't seem to get it to start from the beginning again. Any help would be appreciated.

@warkolm any thoughts on this? sorry to mention you, i just really need help with this one and i feel like i've tried everything and read all the docs.

Edit: Looks like the issue was that I was actually clearing the queue, but the warning was because the DLQ pipeline was not resolving the mapping issues for the documents. When I read the warning log message I thought it was coming from the DLQ pipeline the first timeit tried to process the event.

It would be awesome if it were more verbose. All I saw were Dead Letter Queue warnings. I didn't see any of the usual "Elasticsearch couldn't index..." warnings. I'm not sure why those weren't included. Does DLQ being turned on suppress those warnings or something?


I'm also seeing later on that the DLQ max size limit has been reached. How can I work around this? Is the size limit for each file (1.log, 2.log, etc.) or for the entire pipeline directory?

(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.