Hi all,
I'm working with Logstash 5.5 and the new Feature DQL.
I've installed the plugin version 1.0.5 for LS 5.5:
/usr/share/logstash/bin/logstash-plugin list --verbose
...
logstash-input-dead_letter_queue (1.0.5)
logstash-input-elasticsearch (4.0.4)
...
This is the logstash.yml:
# ------------ Dead-Letter Queue Settings --------------
# Flag to turn on dead-letter queue.
#
dead_letter_queue.enable: true
#
# If using dead_letter_queue.enable: true, the directory path where the data files will be stored.
# Default is path.data/dead_letter_queue
#
path.dead_letter_queue: /data/logstashData/dead_letter_queue
In the dead_letter_queue dir, I have ten empty files (1.log ... 10.log) and in the same logstash pipeline I have 2 inputs:
Kafka
DQL
input {
kafka {
bootstrap_servers => "...:9092,...:9092"
topics_pattern => "logstash-55*"
consumer_threads => 3
decorate_events => true
codec => "json"
}
dead_letter_queue {
path => "/data/logstashData/dead_letter_queue"
commit_offsets => true
codec => "json"
}
}
In order to automatically handle dead events, because I could have some different mapping fields in JSON to ingest, I use a "log_version" field that is incremented by a Ruby script in the pipeline that determines the name of the index (ex: logstash-*-VERSION-YYYY.MM.DD).
In this way, if an event could not be indexed, It's put into the DLQ with version=1, then it's read by DLQ input and the version is incremented by 1 (version=2) and so on.
Now, in Elasticsearch I have this exception: MapperParsingException: object mapping for [mapping_request.payload] tried to parse field [payload] as object, but found a concrete value.
I think that this event must be in the DLQ but nothing is changed.
Logstash Pipeline API:
"inputs" : [ {
"id" : "0fc849ea2666cb7428155be4777136ae64990ef7-2",
"events" : {
"out" : 0,
"queue_push_duration_in_millis" : 0
},
"name" : "dead_letter_queue"
}, {
"id" : "0fc849ea2666cb7428155be4777136ae64990ef7-1",
"events" : {
"out" : 15209085,
"queue_push_duration_in_millis" : 2000258
},
"name" : "kafka"
} ],
How can I verify that DLQ is well-configured?
Thank you.
