I'm using logstash to forward events in JSON format to Elasticsearch to enable adhoc log analysis. I need to identify and review the events/log messages that fail to be indexed. ("Could not index event to Elasticsearch", most due to parsing errors, dynamic mapping mismatches in the data.)
Is there a way to configure the error message to contain the source file and offset, so I can link the error to the specific event?
Alternately, "Trace" level debugging provides the level of detail required. (The error message and the event.) Is there a way to provide the trace level detail only for those events that fail to index? (Ie, avoid that level of logging for the millions of events that are successfully indexed.) Or otherwise save the error message and event to a file?