Questions about dead letter queue

Hi @tgxworld,

Currently, the DLQ logs do not get deleted automatically - this is a feature that will be added in a future Logstash release.

A new DLQ log file is created either a) when Logstash is started up, or b) when a new event would cause the log file to be larger than the 'maximum segment size' - this is currently set to 10MB.

To view the Dead Letter Queue there is a dead letter queue input plugin. This can be processed like any other Logstash input - initially you can pipe to stdout, or similar, using the rubydebug filter (with metadata enabled) to determine the errors. Then you can use the mutate filter to remove any mapping errors (as shown here)

The commit_offset flag when set to true will cause the last processed DLQ entry to be stored in the .sincedb file stored in the ${path.data}/plugins/inputs/dead_letter_queue/${pipeline.id} folder (pipeline_id will be main unless it has been manually set. If the commit_offset flag is set to false, these offsets will not be stored.

Thanks!

Rob

2 Likes