Ingest pipeline is giving weird response to timeformat change

Elastic Stack Elasticsearch
1

Hi!

I'm stuck... I'd tested this pipeline in local, and now in a remote server (using the same docker image, mind you, v6.3.1), it's not behaving as I'd expected...

{
  "description": "Takes the eventTime field and turns it into a date field",
  "processors": [
    {
      "date": {
        "field": "eventTime",
        "target_field": "@timestamp",
        "formats": [
          "YYYY-MM-DD'T'HH:mm:ssZ",
          "YYYY/MM/DD HH:mm:ssZ"
        ]
      }
    }
  ],
  "on_failure": [
    {
      "set": {
        "field": "_index",
        "value": "failed-{{_index}}"
      }
    },
    {
      "set": {
        "field": "error",
        "value": "{{_ingest.on_failure_message}}"
      }
    }
  ]
}

Sample input:

{"eventType":"autosuggest","user":127903,"eventTime":"2018-08-29T00:41:35+1000","searchTerm":"swee"}
{"eventType":"autosuggest","user":127903,"eventTime":"2018-08-29T00:41:35+1000","searchTerm":"sweet "}
{"eventType":"autosuggest","user":127903,"eventTime":"2018-08-29T00:41:35+1000","searchTerm":"sweet so"}

Sample output from the index:

As you can see below, it takes an eventtime value of 2018-08-29T01:18:36+1000, and then @timestamp turns into 2018-01-28T15:18:36.000Z.

{
    "took": 4,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": 29158,
        "max_score": 1,
        "hits": [
            {
                "_index": "search-metrics-filebeat-6.3.2-2018.08.28",
                "_type": "doc",
                "_id": "kMgegWUBozxkfJZkg-yB",
                "_score": 1,
                "_source": {
                    "offset": 16073720,
                    "prospector": {
                        "type": "log"
                    },
                    "source": "/usr/share/filebeat/logs/event_metrics_1535467082683.log",
                    "eventType": "autosuggest",
                    "tags": [
                        "json",
                        "search-events"
                    ],
                    "input": {
                        "type": "log"
                    },
                    "@timestamp": "2018-01-28T15:18:36.000Z",
                    "searchTerm": "chic",
                    "beat": {
                        "hostname": "6c703e3fceb2",
                        "name": "6c703e3fceb2",
                        "version": "6.3.2"
                    },
                    "eventTime": "2018-08-29T01:18:36+1000",
                    "host": {
                        "name": "6c703e3fceb2"
                    },
                    "user": 141139
                }
            },
            {
                "_index": "search-metrics-filebeat-6.3.2-2018.08.28",
                "_type": "doc",
                "_id": "m8gegWUBozxkfJZkg-yB",
                "_score": 1,
                "_source": {
                    "offset": 16074859,
                    "prospector": {
                        "type": "log"
                    },
                    "eventType": "autosuggest",
                    "source": "/usr/share/filebeat/logs/event_metrics_1535467082683.log",
                    "tags": [
                        "json",
                        "search-events"
                    ],
                    "input": {
                        "type": "log"
                    },
                    "@timestamp": "2018-01-28T15:18:36.000Z",
                    "searchTerm": "white su",
                    "eventTime": "2018-08-29T01:18:36+1000",
                    "beat": {
                        "hostname": "6c703e3fceb2",
                        "name": "6c703e3fceb2",
                        "version": "6.3.2"
                    },
                    "host": {
                        "name": "6c703e3fceb2"
                    },
                    "user": 103776
                }
            }
        ]
    }
}

I've got absolutely no idea why this isn't working... Any chance of some help?
Thanks!

2

If I remove the pipeline's processor - so it's just an empty array - the @timestamp goes to its original value of "ingested time". So it's definitely something strange happening with the processor. :confused:

3

Ok. I figured out why...

I was using DD instead of dd.
DD = day of the year
dd = day of the month

Hopefully nobody else runs into such an embarrassing problem!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.