Log Events/Records that Cause Exceptions

[swsw]

Hello,

How do I log the events/records that come into Logstash that cause exceptions (specifically illegal_argument_exception when putting data into Elasticsearch from Logstash).

Situation: I am using Logstash and Elasticsearch 5.3. I have data coming in that I don't have much control over. I have a schema set on the index that Logstash is putting the data into. Sometimes data comes in that doesn't match the schema. I get the following record in the Logstash log:

[2017-07-24T19:52:38,086][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"active-legacy", :_type=>"type", :_routing=>nil}, 2017-07-24T22:56:54.076Z xxxx %{message}], :response=>{"index"=>{"_index"=>"index-legacy", "_type"=>"type", "_id"=>"AV13BGKU0yA-YGydx47G", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [data_model.fields.utc_time]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"2017-07-24 22:56:54.076\" is malformed at \" 22:56:54.076\""}}}}}

What I would like is an option to log the records that came in that led to that exception, so that as I tweak my logstash config, I can test to make sure that my changes work.

Something like:

output {
    elasticsearch {
        hosts => ...
        index => ....
        log_on_failure => "/tmp/es_debug.log"
    }
}

Or if instead there was a global logstash option that logs records when exceptions occur?

Does anything like this exist?

Thanks

[magnusbaeck (Magnus Bäck)]

Have you look at the dead letter queue feature (introduced in 5.5): https://www.elastic.co/guide/en/logstash/current/dead-letter-queues.html

[swsw]

That's exactly wat I need. Will have to look into upgrading =)