How to track down a document based on a Logstash error

Elastic Stack Logstash
1

I'm getting an error from Logstash that looks like this:

Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"np-audit-7.4.0-2019.11.14", :_type=>"_doc", :_routing=>nil}, #<LogStash::Event:0x5cb25869>], :response=>{"index"=>{"_index"=>"data-7.4.0-2019.11.14", "_type"=>"_doc", "_id"=>"1NGxcG4B3XBDJAVLu464", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [data.added] of type [text] in document with id '1NGxcG4B3XBDJAVLu464'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:3136"}}}}}

I'm trying to figure out how to get back to the document that caused the error (shipped from filebeat) based on the document ID 1NGxcG4B3XBDJAVLu464. We have filebeat setup on a number of servers shipping those logs, I'm not sure if filebeat is setting that ID or Logstash creates it once it receives the document. Bottomline, I'd like to see what the document looks like to see what's wrong with it and why it's causing an error, but I have no clue how to work my way back to the document/log record.

2

Use a DLQ?

3

We're putting DLQ in place now, but it was not setup when the error happened. Is that the only option?

4

It's the only one I can think of.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.