Debugging logstash indexing issue - trace output on index failure?

Elastic Stack Logstash
1

I'm using logstash to forward events in JSON format to Elasticsearch to enable adhoc log analysis. I need to identify and review the events/log messages that fail to be indexed. ("Could not index event to Elasticsearch", most due to parsing errors, dynamic mapping mismatches in the data.)

Is there a way to configure the error message to contain the source file and offset, so I can link the error to the specific event?

Alternately, "Trace" level debugging provides the level of detail required. (The error message and the event.) Is there a way to provide the trace level detail only for those events that fail to index? (Ie, avoid that level of logging for the millions of events that are successfully indexed.) Or otherwise save the error message and event to a file?

2

That is what dead letter queues do. If the reason stored in the DLQ is not detailed enough you could run an instance with a DLQ input and an elasticsearch output with trace level logging to get the detail without getting trace level logging for events that are indexed successfully.

3

Perfect! I'm just starting to learn about Elasticsearch, really appreciate your help!

Bill

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.