# Pipeline date\_time parser failure beats me (sorry for the pun :)

**URL:** <https://discuss.elastic.co/t/pipeline-date-time-parser-failure-beats-me-sorry-for-the-pun/343380>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 19, 2023, 2:47pm UTC](https://discuss.elastic.co/t/pipeline-date-time-parser-failure-beats-me-sorry-for-the-pun/343380 "2023-09-19T14:47:08Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![stefws](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stefws/32/6442_2.png) [@stefws](https://discuss.elastic.co/u/stefws)\
**Post date:** [September 19, 2023, 2:47pm UTC](https://discuss.elastic.co/t/pipeline-date-time-parser-failure-beats-me-sorry-for-the-pun/343380/1 "2023-09-19T14:47:08Z")

</div>

I'm ingesting Redhat AMQ log with filebeat, only filebeat claims the ingest pipeline fails to parse date time of every event. But testing a sample event/document from fiebeat log, pipeline works fine, that _beats_ me. Hints appreciated, TIA!

filebeat log snippet:

```auto
\"reason\":\"failed to parse date field [2023-09-18 20:39:54,127] with format [strict_date_optional_time||epoch_millis]\",\"caused_by\":{\"type\":\"date_time_parse_exception\",\"reason\":\"date_time_parse_exception: Failed to parse with all enclosed parsers\"}}}, dropping event!","service.name":"filebeat","ecs.version":"1.6.0"

```

Testing above failed doc in the pipeline is successful thou:

 ![Screenshot 2023-09-19 at 16.39.00](https://us1.discourse-cdn.com/elastic/original/3X/f/5/f5edfaedfecf6e4758c8ecefa93d22c6b49892ff.png)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [September 19, 2023, 3:00pm UTC](https://discuss.elastic.co/t/pipeline-date-time-parser-failure-beats-me-sorry-for-the-pun/343380/2 "2023-09-19T15:00:42Z")

</div>

Hi @stefws

The issue is when the data is actually written to the index that is when the schema / format is validated which does not fit `strict_date_optional_time||epoch_millis`

The pipeline can parse but the source data but can not be written

What I see is 2 things 1 that date is in neither of the formats above  
and you may have this is a "locale" issue with the date format the `,` in the millis instead of `.` though I am not completely sure if that will actually be an issue.

Take a look [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html)

Hmmm...The fastest way to fix this is

a) Fix the timestamp format at the source to match one of the OOTB formats  
b) Add your format to the template filbeat template

---

<div class="post-metadata">

**Author:** ![stefws](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stefws/32/6442_2.png) [@stefws](https://discuss.elastic.co/u/stefws)\
**Post date:** [September 19, 2023, 3:09pm UTC](https://discuss.elastic.co/t/pipeline-date-time-parser-failure-beats-me-sorry-for-the-pun/343380/3 "2023-09-19T15:09:45Z")

</div>

Thanks for hinting... or enhance pipeline to convert timestamp further 😉

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [September 19, 2023, 3:53pm UTC](https://discuss.elastic.co/t/pipeline-date-time-parser-failure-beats-me-sorry-for-the-pun/343380/4 "2023-09-19T15:53:58Z")

</div>

Yep good idea you could use the `date` processor

> **[Date processor | Elasticsearch Guide \[8.10\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/date-processor.html)**

---

<div class="post-metadata">

**Author:** ![stefws](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stefws/32/6442_2.png) [@stefws](https://discuss.elastic.co/u/stefws)\
**Post date:** [September 19, 2023, 5:30pm UTC](https://discuss.elastic.co/t/pipeline-date-time-parser-failure-beats-me-sorry-for-the-pun/343380/5 "2023-09-19T17:30:51Z")

</div>

Turned out I was missing just that, like I've got on other log file pipelines, w/simple date format wo/timezone info, a Date processor like this:

 ![Screenshot 2023-09-19 at 17.24.51](https://us1.discourse-cdn.com/elastic/original/3X/1/7/17d969ebf36e1c76a295066d0299157825a46c32.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 17, 2023, 7:30pm UTC](https://discuss.elastic.co/t/pipeline-date-time-parser-failure-beats-me-sorry-for-the-pun/343380/6 "2023-10-17T19:30:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.