Date processor - Pipeline

Elastic Stack Elasticsearch
1

Hi guys.
I'm trying to add the date processor to my index and at first it seemed to be working but then I noticed very strange result that I don't understand and can't find anywhere explanation for this behavior.
I have a field with long-millis timestamps and I want it to be translated into human date string.

So I made such pipeline processor:

PUT _ingest/pipeline/long_to_date_converter
{
  "processors" : [
    {
      "date": {
        "field" : "timestamp",
        "formats": ["epoch_millis"],
        "target_field": "timestamp1"
      }
    }
  ]
}

But then I have got this strange result, all my timestamps were converted into same value 2022-01-01T00:00:00.000Z

If I run the simulate on the pipeline then I get this result:

{
  "docs" : [
    {
      "doc" : {
        "_index" : "index",
        "_type" : "_doc",
        "_id" : "id",
        "_source" : {
          "timestamp1" : "2022-01-01T00:00:00.000Z",
          "timestamp" : "1581951442000"
        },
        "_ingest" : {
          "timestamp" : "2022-02-18T06:46:18.0853227Z"
        }
      }
    }
  ]
}

Can anyone explain why it happens and how can I achieve the right behavior? I'm using ES version 7.10.2

Timestamp I used in the example (1581951442000) in human date string should be Mon Feb 17 2020 14:57:22

2

Hi @p2w2 Welcome to the community..BTW we are not all guys :slight_smile:

Per the docs here

formats yes -
An array of the expected date formats. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

Looks like you are using the formats that are used for a mapping not the ingest processor, easy to confused AND they should probably be the consistent at some point

PUT _ingest/pipeline/long_to_date_converter
{
  "processors" : [
    {
      "date": {
        "field" : "timestamp",
        "formats": ["UNIX_MS"], <!--- Correct Format
        "target_field": "timestamp1"
      }
    }
  ]
}

POST _ingest/pipeline/long_to_date_converter/_simulate
{
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "timestamp": "1581951442000"
      }
    }
  ]
}

result

{
  "docs" : [
    {
      "doc" : {
        "_index" : "index",
        "_type" : "_doc",
        "_id" : "id",
        "_source" : {
          "timestamp1" : "2020-02-17T14:57:22.000Z",
          "timestamp" : "1581951442000"
        },
        "_ingest" : {
          "timestamp" : "2022-02-19T17:00:45.381877547Z"
        }
      }
    }
  ]
}
3

Sorry for the "guys", my bad.
And thanks for the reply and the documentation reference.
Feels so stupid for not reading basic guidance. I wonder now where did I copied this epoch_millis

1 Like
4

All good. Glad you got it solved. It's what we're here for!

Is defined as a pattern for a date type in the mapping So you probably saw it here

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.