Hi,
I'm trying to process events written to a dlq due to mapping conflicts. I'm not trying to preprocess any of those messages (yet). Instead, am just reading them in and writing them to a new index name. However, I get the error:
[WARN ] 2022-12-06 15:04:19.778 [[out-dlq]>worker6] DeadLetterQueueWriter - Event previously submitted to dead letter queue. Skipping...
Why is this? The new index doesn't yet exist and so there is no data to conflict with. Logstash version is 8.4.3 and test config looks like this:
input {
dead_letter_queue {
path => "/path_to_dlq/dead_letter_queue"
pipeline_id => "out-es"
commit_offsets => true
id => "dlq-out"
type => "dlq"
}
}output {
elasticsearch {
hosts => "elasticsearch_endpoint:9200"
index => "logs-dlq-%{+YYYY.MM.dd}"
manage_template => false
document_type => "_doc"
pipeline => "ingestion_pipeline"
}
}
Note: we're using a pipelined configuration which isn't shown here.
Thx
D