Pipeline date_time parser failure beats me (sorry for the pun :)

[stefws (Steffen Winther Sørensen)]

I'm ingesting Redhat AMQ log with filebeat, only filebeat claims the ingest pipeline fails to parse date time of every event. But testing a sample event/document from fiebeat log, pipeline works fine, that beats me. Hints appreciated, TIA!

filebeat log snippet:

\"reason\":\"failed to parse date field [2023-09-18 20:39:54,127] with format [strict_date_optional_time||epoch_millis]\",\"caused_by\":{\"type\":\"date_time_parse_exception\",\"reason\":\"date_time_parse_exception: Failed to parse with all enclosed parsers\"}}}, dropping event!","service.name":"filebeat","ecs.version":"1.6.0"

Testing above failed doc in the pipeline is successful thou:

[stephenb (Stephen Brown)]

Hi @stefws

The issue is when the data is actually written to the index that is when the schema / format is validated which does not fit strict_date_optional_time||epoch_millis

The pipeline can parse but the source data but can not be written

What I see is 2 things 1 that date is in neither of the formats above
and you may have this is a "locale" issue with the date format the , in the millis instead of . though I am not completely sure if that will actually be an issue.

Take a look here

Hmmm...The fastest way to fix this is

a) Fix the timestamp format at the source to match one of the OOTB formats
b) Add your format to the template filbeat template

[stefws (Steffen Winther Sørensen)]

Thanks for hinting... or enhance pipeline to convert timestamp further

[stephenb (Stephen Brown)]

Yep good idea you could use the date processor

Date processor | Elasticsearch Guide [8.10] | Elastic

[stefws (Steffen Winther Sørensen)]

Turned out I was missing just that, like I've got on other log file pipelines, w/simple date format wo/timezone info, a Date processor like this: