# Parsing multiple date types from message field with ingest node

**URL:** <https://discuss.elastic.co/t/parsing-multiple-date-types-from-message-field-with-ingest-node/149821>\
**Category:** Elasticsearch\
**Created:** [September 25, 2018, 12:34pm UTC](https://discuss.elastic.co/t/parsing-multiple-date-types-from-message-field-with-ingest-node/149821 "2018-09-25T12:34:27Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![dlazarov](https://avatars.discourse-cdn.com/v4/letter/d/bb73d2/32.png) [@dlazarov](https://discuss.elastic.co/u/dlazarov)\
**Post date:** [September 25, 2018, 12:34pm UTC](https://discuss.elastic.co/t/parsing-multiple-date-types-from-message-field-with-ingest-node/149821/1 "2018-09-25T12:34:27Z")

</div>

I have the following case. I have setup Filebeat to send the logs straight to Elasticsearch since there is no need for significant log parsing. I am receiving logs from different services and in the message field the date format is different for each service. The version I'm using is 5.6

I have setup a pipeline with grok and date processors like this:

```
PUT _ingest/pipeline/test_pipeline
{
  "description": "timestamp test",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{SYSLOGTIMESTAMP:systime}", "%{EXIM_DATE:eximdate}"]
      },
      "date": {
        "field": "systime",
        "formats": ["MMM dd HH:mm:ss"],
        "ignore_failure": true
      },
      "date": {
        "field": "eximdate",
        "formats": ["yyyy-MM-dd HH:mm:ss"],
        "ignore_failure": true
      }, 
      "remove": {
        "field": ["systime", "eximdate"],
        "ignore_failure": true
      }
    }
  ]
}

```

I have checked the patterns individually and they both work, the issue here is that when I'm simulation with messages of each type, the `@timestamp` is updated only for the second pattern.

Here are the messages I am using to simulate the pipeline:

- First pattern

- Second pattern

Here are the outputs for both types of messages

- First pattern simulation output

- Second pattern simulation output

You can clearly see that the `@timestamp` field shows up only on the second kind of pattern. And yet, they work perfectly fine if there is only one pattern and one date processor in the pipeline.

---

<div class="post-metadata">

**Author:** ![jakelandis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jakelandis/32/36163_2.png) [@jakelandis](https://discuss.elastic.co/u/jakelandis)\
**Post date:** [September 25, 2018, 1:39pm UTC](https://discuss.elastic.co/t/parsing-multiple-date-types-from-message-field-with-ingest-node/149821/2 "2018-09-25T13:39:45Z")

</div>

Try this:

```auto
PUT _ingest/pipeline/test_pipeline
{
  "description": "timestamp test",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          "%{SYSLOGTIMESTAMP:systime}",
          "%{EXIM_DATE:eximdate}"
        ]
      }
    },
    {
      "date": {
        "field": "systime",
        "formats": [
          "MMM dd HH:mm:ss"
        ],
        "ignore_failure": true
      }
    },
    {
      "date": {
        "field": "eximdate",
        "formats": [
          "yyyy-MM-dd HH:mm:ss"
        ],
        "ignore_failure": true
      }
    },
    {
      "remove": {
        "field": [
          "systime",
          "eximdate"
        ],
        "ignore_failure": true
      }
    }
  ]
}

```

^^ note the extra `{` and `}` around each processor. New versions (not sure exactly when) don't allow the format you posted and will error when trying to create the pipeline.

EDIT: fixed example

---

<div class="post-metadata">

**Author:** ![dlazarov](https://avatars.discourse-cdn.com/v4/letter/d/bb73d2/32.png) [@dlazarov](https://discuss.elastic.co/u/dlazarov)\
**Post date:** [September 26, 2018, 6:55am UTC](https://discuss.elastic.co/t/parsing-multiple-date-types-from-message-field-with-ingest-node/149821/3 "2018-09-26T06:55:57Z")

</div>

That did it, thanks a bunch. It would have taken me quite a while to figure that out myself.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 24, 2018, 6:56am UTC](https://discuss.elastic.co/t/parsing-multiple-date-types-from-message-field-with-ingest-node/149821/4 "2018-10-24T06:56:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.