Difference between UNIX_MS and epoch_millis in date processor?

Elastic Stack Elasticsearch
1

Hi, I use ingest pipeline to add @timestamp field automatically from a exist date type field.
My exist date type field is called uploadTime and it's date format is epoch_millis.

In the processor, I firstly use epoch_millis as formats and output_format, but I find new @timestamp value is not equal to uploadTime value.
Then I use UNIX_MS as formats and epoch_millis as output_format, @timestamp is equal to uploadTime.

So my question is what's the difference between UNIX_MS and epoch_millis in date processor?
Thanks.

Infomations:

1. Elastic Stack version is 8.4.

2. UNIX_MS test case:

processor:

PUT _ingest/pipeline/test-pipeline
{
  "processors": [
    {
      "date": {
        "field": "uploadTime",
        "formats": [
          "UNIX_MS"
        ],
        "target_field": "@timestamp",
        "output_format": "epoch_millis",
        "on_failure": [
          {
            "append": {
              "field": "error.message",
              "value": [
                "{{ _ingest.on_failure_message }}"
              ]
            }
          }
        ]
      }
    }
  ]
}

test doc:

[
  {
    "_score": 1,
    "_source": {
      "uploadTime": 1683701716065
    }
  }
]

test result:

{
  "_id": "_id",
  "_index": "_index",
  "_version": "-3",
  "_source": {
    "uploadTime": 1683701716065,
    "@timestamp": "1683701716065"
  },
  "_ingest": {
    "pipeline": "_simulate_pipeline",
    "timestamp": "2023-05-10T08:19:25.595621959Z"
  }
}

3. epoch_millis test case:

processor:

PUT _ingest/pipeline/test-pipeline
{
  "processors": [
    {
      "date": {
        "field": "uploadTime",
        "formats": [
          "epoch_millis"
        ],
        "target_field": "@timestamp",
        "output_format": "epoch_millis",
        "on_failure": [
          {
            "append": {
              "field": "error.message",
              "value": [
                "{{ _ingest.on_failure_message }}"
              ]
            }
          }
        ]
      }
    }
  ]
}

test doc is same:

[
  {
    "_score": 1,
    "_source": {
      "uploadTime": 1683701716065
    }
  }
]

test result:

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_version": "-3",
        "_source": {
          "uploadTime": 1683701716065,
          "@timestamp": "1672531200065"
        },
        "_ingest": {
          "timestamp": "2023-05-10T08:23:40.197004932Z"
        }
      }
    }
  ]
}
2

Thanks for the detailed examples. This really looks like a bug in the handling of epoch_millis to me. I created Fixing DateProcessor when the format is epoch_millis by masseyke · Pull Request #95996 · elastic/elasticsearch · GitHub as a potential fix for it, but will discuss with the team to see if that's the right way to go.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.