Hi there, we've recently started looking into setting up a dead_letter_queue pipeline at Discourse but I had some questions which wasn't immediately clear from the documentation.
Does the DLQ log files get deleted at all? From what I'm seeing currently, the log files remain on disk even after it has been processed by a pipeline. Also, when does logstash decide to create a new DLQ log file? (1.log, 2.log etc...)
What is the recommended practice for setting up a dead_letter_queue pipeline? Since the DQL log files are not really human readable, is it recommend to setup the pipeline containing by a stdout and elasticsearch output?
Is it recommended to set commit_offset to true for monitoring purposes and then flick it to false once you're ready to fix the underlying bug?
When commit_offset is true, where does logstash save the state?
Currently, the DLQ logs do not get deleted automatically - this is a feature that will be added in a future Logstash release.
A new DLQ log file is created either a) when Logstash is started up, or b) when a new event would cause the log file to be larger than the 'maximum segment size' - this is currently set to 10MB.
To view the Dead Letter Queue there is a dead letter queue input plugin. This can be processed like any other Logstash input - initially you can pipe to stdout, or similar, using the rubydebug filter (with metadata enabled) to determine the errors. Then you can use the mutate filter to remove any mapping errors (as shown here)
The commit_offset flag when set to true will cause the last processed DLQ entry to be stored in the .sincedb file stored in the ${path.data}/plugins/inputs/dead_letter_queue/${pipeline.id} folder (pipeline_id will be main unless it has been manually set. If the commit_offset flag is set to false, these offsets will not be stored.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.