I am using version 7.1.0 Elastic stack to ingest logs into Elasticsearch.
When procssing the data, I encounter this ERROR message several times:
[2019-12-19T12:30:10,332][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"https://172.31.xx.xx:443/_bulk"}
After setting the log level to DEBUG mode in Logstash, it seems that I am trying to ingest files that do not have an ID field.So, this failure makes senses to me. However, since I am getting a 400 error, I thought this would be a good chance to use the dead_letter_queue to log the offending messages.
I configured my logstash.yml to enable DLQ per the documentation: https://www.elastic.co/guide/en/logstash/current/dead-letter-queues.html#dlq-example
logstash.yml
......
dead_letter_queue.enable: true
dead_letter_queue.max_bytes: 1024mb
path.dead_letter_queue: "/usr/share/logstash/data/dead_letter_queue"
The correct directory is created for my pipeline in the dead_letter_queue directory.
But the *.log files have nothing in them. I expected to see the error messages or offending log entries.
Have I misunderstood the DLQ?
Thank you!