Hi, I use ingest pipeline to add @timestamp field automatically from a exist date type field.
My exist date type field is called uploadTime and it's date format is epoch_millis.
In the processor, I firstly use epoch_millis as formats and output_format, but I find new @timestamp value is not equal to uploadTime value.
Then I use UNIX_MS as formats and epoch_millis as output_format, @timestamp is equal to uploadTime.
So my question is what's the difference between UNIX_MS and epoch_millis in date processor?
Thanks.
Infomations:
1. Elastic Stack version is 8.4.
2. UNIX_MS test case:
processor:
PUT _ingest/pipeline/test-pipeline
{
"processors": [
{
"date": {
"field": "uploadTime",
"formats": [
"UNIX_MS"
],
"target_field": "@timestamp",
"output_format": "epoch_millis",
"on_failure": [
{
"append": {
"field": "error.message",
"value": [
"{{ _ingest.on_failure_message }}"
]
}
}
]
}
}
]
}
test doc:
[
{
"_score": 1,
"_source": {
"uploadTime": 1683701716065
}
}
]
test result:
{
"_id": "_id",
"_index": "_index",
"_version": "-3",
"_source": {
"uploadTime": 1683701716065,
"@timestamp": "1683701716065"
},
"_ingest": {
"pipeline": "_simulate_pipeline",
"timestamp": "2023-05-10T08:19:25.595621959Z"
}
}
3. epoch_millis test case:
processor:
PUT _ingest/pipeline/test-pipeline
{
"processors": [
{
"date": {
"field": "uploadTime",
"formats": [
"epoch_millis"
],
"target_field": "@timestamp",
"output_format": "epoch_millis",
"on_failure": [
{
"append": {
"field": "error.message",
"value": [
"{{ _ingest.on_failure_message }}"
]
}
}
]
}
}
]
}
test doc is same:
[
{
"_score": 1,
"_source": {
"uploadTime": 1683701716065
}
}
]
test result:
{
"docs": [
{
"doc": {
"_index": "_index",
"_id": "_id",
"_version": "-3",
"_source": {
"uploadTime": 1683701716065,
"@timestamp": "1672531200065"
},
"_ingest": {
"timestamp": "2023-05-10T08:23:40.197004932Z"
}
}
}
]
}