I'm recently encountering problem with my ingestion pipeline, where part of if tries to parse a date. If it fails to do so, it should continue and not create the fields time_log and rest_message.
This works for any message that does not contain a date and the log is being parsed.
However, for some datetime formats it crashes and does not event create an entry at all. E.g. 2021-01-12 13:39:28.620+0100 is being parsed and 2021-02-09 22:15:28 is not.
The error message that I'm getting is:
""type":"illegal_argument_exception","reason":"failed to parse date field [2021-02-09 22:15:28] with format [strict_date_optional_time||epoch_millis]","caused_by":{"type":"date_time_parse_exception","reason":"date_time_parse_exception: Failed to parse with all enclosed parsers""
I thought that this case is just ignored, however is seems to parse the date but not to create the entry. How can I make sure that either:
the date is being parsed
the pipeline fails but does create a document
?
This is the relevant part of my pipeline:
{
"grok": {
"field": "message",
"patterns": [
"%{TIMESTAMP_ISO8601:time_log:data} %{GREEDYDATA:rest_message}"
],
"ignore_failure" : true
}
},
Thanks.
It's better if you can format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you
It looks like this is the issue;
As that does not match 2021-02-09 22:15:28.
You should add another pattern in the formats field that matches that, and you should be fine.
thanks!
I will try this as soon as possible. Apart from that: is there any way to catch the error or notify me?
So let's say there will be another unsupported time format.
I thought that the error would be caught by the on_failure section?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.