Hello everybody,
I'm recently encountering problem with my ingestion pipeline, where part of if tries to parse a date. If it fails to do so, it should continue and not create the fields time_log
and rest_message
.
This works for any message that does not contain a date and the log is being parsed.
However, for some datetime formats it crashes and does not event create an entry at all. E.g.
2021-01-12 13:39:28.620+0100
is being parsed and 2021-02-09 22:15:28
is not.
The error message that I'm getting is:
""type":"illegal_argument_exception","reason":"failed to parse date field [2021-02-09 22:15:28] with format [strict_date_optional_time||epoch_millis]","caused_by":{"type":"date_time_parse_exception","reason":"date_time_parse_exception: Failed to parse with all enclosed parsers""
I thought that this case is just ignored, however is seems to parse the date but not to create the entry. How can I make sure that either:
- the date is being parsed
- the pipeline fails but does create a document
?
This is the relevant part of my pipeline:
{
"grok": {
"field": "message",
"patterns": [
"%{TIMESTAMP_ISO8601:time_log:data} %{GREEDYDATA:rest_message}"
],
"ignore_failure" : true
}
},
all the best,
Glenn