Date format error with "epoch_millis||epoch_second"

Elastic Stack Elasticsearch
1

Hi all....
Here i am trying to define mapping for a field which stores epoch date values in it.

PUT my_index11
{
  "mappings": {
    "my_type": {
      "properties": {
        "date": {
          "type": "date",
           "format": "epoch_millis||epoch_second"
        }
      }
    }
  }
}

where in my mapping i have defined two Built In Formats they are epoch_millis||epoch_second
but the problem is when i insert a document with date value in epoch_millis

PUT my_index11/my_type/1
{ "date": 1505894323000 }

i am able to find a doc 1 in discovery page of kibana but when i insert a document with date value in epoch_second

PUT my_index11/my_type/2
{ "date": 1420070400 }

i have not find second doc i.e, doc 2 in kibana discovery page but i able to see it in index

Please help me with this

Thank You.

2

I suspect the second document is being indexed with the date "Saturday, 17 January 1970 10:27:50.400". The reason for this is that dates in the epoch_second format also look like they are in epoch_milli since neither format is fixed width. When Elasticsearch indexes a document it tries the formats in the order you specify in the mapping and stops when it reaches the first format that succeeds which means that it will first try epoch_milli that will always match succeed since 1420070400 is a valid epoch_milli timestamp (though not the timestamp you intend).

I think you'll need to solve this upstream in your indexing pipeline and normalise all your dates to epoch_millis. Presumably a particular data source is consistent with the format it sends so it may just be that you need to add logic to your indexing application that converts the dates to epoch_milli if it is from the data sources that send epoch_second

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.