Error in processing events from DLQ

[pandeesh (Pandeeswaran Bhoopathy)]

All, I am trying to expose the raw events in a index dedicated to DLQ.

These are the fields I would like to expose:

index_name (where it was unable to perform originally)
error_reason(exception)
raw_event(_source)
document_id

Here's my current DLQ configuration:

input {
dead_letter_queue {
path => "/var/lib/logstash/dead_letter_queue"
commit_offsets => true
}
}
output {
elasticsearch {
document_type => "dlq"
codec => rubydebug { metadata => true }
hosts => ["host"]
index => "dlq-test-%{+YYYY.MM.DD}"
}

}

But the events are still failing to index with the same reason as original pipeline. I am getting:

[2018-03-29T11:55:18,463][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"dlq-test-2018.03.88", :_type=>"dlq", :_routing=>nil}, 2018-03-29T18:54:29.400Z xxx-prod-10-3-10-119 %{message}], :response=>{"index"=>{"_index"=>"dlq-test-2018.03.88", "_type"=>"dlq", "_id"=>"AWJzHTs9B5EI", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [fields.xxx.xxx]tried to parse field [properties] as object, but found a concrete value"}}}}

even with codec as JSON, I am seeing the same behavior. Either with json or rubydebug codec, i am unable to see the error message/exception in the dlq-test* index in kibana.

if I want to only retrieve error, index_name, document_id and raw_event, is this the correct codec to use? thanks for your help.

[pandeesh (Pandeeswaran Bhoopathy)]

This looks like promising: https://lukewaite.ca/posts/2017/10/13/viewing-logstash-dlq-in-kibana.html
exploring this. thanks

[pandeesh (Pandeeswaran Bhoopathy)]

However I have found another problem while implementing this. my dlq pipeline is similar to the one mentioned here:kewaite.ca/posts/2017/10/13/viewing-logstash-dlq-in-kibana.html

input {
dead_letter_queue {
path => "/var/lib/logstash/dead_letter_queue"
commit_offsets => true
}
}
filter {
# First, we must capture the entire event, and write it to a new
# field; we'll call that field failed_message
ruby {
code => "event.set('failed_message', event.to_json())"
}
# Next, we prune every field off the event except for the one we've
# just created. Note that this does not prune event metadata.
prune {
whitelist_names => [ "^failed_message$" ]
}
# Next, convert the metadata timestamp to one we can parse with a
# date filter. Before conversion, this field is a Logstash::Timestamp.
# http://www.rubydoc.info/gems/logstash-core/LogStash/Timestamp
ruby {
code => "event.set('timestamp', event.get('[@metadata][dead_letter_queue][entry_time]').toString())"
}
# Apply the date filter.
date {
match => [ "timestamp", "ISO8601" ]
}
# Pull useful information out of the event metadata provided by the dead
# letter queue, and add it to the new event.
mutate {
add_field => {
"message" => "%{[@metadata][dead_letter_queue][reason]}"
"plugin_id" => "%{[@metadata][dead_letter_queue][plugin_id]}"
"plugin_type" => "%{[@metadata][dead_letter_queue][plugin_type]}"
}
}
}
output {
elasticsearch {
document_type => "dlq"
hosts => ["host"]
index => "dlq-events-%{+YYYY.MM.dd}"
} }

However, when there's no messages/events in the input dlq path, it breaks with ,

[2018-03-29T14:13:57,569][ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined method toString' for nil:NilClass [2018-03-29T14:13:57,747][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: timestamp field is missing>, :backtrace=>["org/logstash/ext/JrubyEventExtLibrary.java:205:in sprintf'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/common.rb:168:in event_action_params'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/common.rb:44:in event_action_tuple'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/common.rb:38:in multi_receive'", "org/jruby/RubyArray.java:2414:in map'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/common.rb:38:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:49:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:434:in output_batch'", "org/jruby/RubyHash.java:1342:in each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:433:in output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:381:in worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:342:in start_workers'"]}

Is there a recommended way to handle this case? It's obvious that dead_letter_queue can be often empty and it has data only sometimes when there are issues. Is there a technique to handle this scenario gracefully without failing all the logstash pipelines running?

I feel,

commit_offsets=> true

is not doing the right thing here. I expect it should not pass any records to filter.

Upon, further investigation it looks like DLQ confg in primary pipeline is generating the single byte file with empty content which triggers this error in the DLQ pipeline. is it a known issue or any workaround for this?

plug-in version:

logstash-input-dead_letter_queue (1.1.2)

This is the dlq config in primary pipeline main:

# ------------ Dead-Letter Queue Settings --------------
 # Flag to turn on dead-letter queue.
 #
 dead_letter_queue.enable: true
 dead_letter_queue.max_bytes: 4g

also, noticed that when I have just a single pipeline without the secondary one DLQ pipeline, it's able to write the proper logs to DLQ directory.
while introducing pipelines.yml with 2 pipelines(primary and DLQ) which overrides the logstash.yml starts causing the single byte issue.

Thanks for your help