Limitation of storing documents with a timestamp far far in the future

Elastic Stack Elasticsearch
1

Hi Community

The @timestamp field of an index template has been set in the format of epoch_seconds.
Unfortunately the application sends the data in epoch_millis (1577983263046).
Hence the json events are stored somewhere in year 04/20/51974.

I've queried the data for results @timestamp higher than today, but I couldn't find events.
From my point of view it looks that elasticsearch skips such documents.

My question: Of course our scenario is an application issue but nevertheless I'm wondering whether there is a limitation storing documents far in the future? Is there?

Thanks

2

No, as long as the value doesn't surpass the largest value that can stored in the underlying data structure, there is no limit on dates in the future.

This example demonstrates the behaviour. Two documents are stored, where one contains the date value as epoch_second, and the other as epoch_millis. The mapping only specifies epoch_second and also basic_date_optional_time. Only date strings in a recognized format can be searched for.

DELETE test-date
PUT test-date
{
  "mappings": {
    "properties": {
      "message": {
        "type": "text"
      },
      "@timestamp": {
        "type": "date",
        "format": "date_optional_time||epoch_second"
      }
    }
  }
}

PUT test-date/_doc/1
{
  "@timestamp": "1577983263046",
  "message": "Hello Millis"
}

PUT test-date/_doc/2
{
  "@timestamp": "1577983263",
  "message": "Hello Seconds"
}

GET test-date/_search
{
  "size": 10, 
  "query": {
    "range": {
      "@timestamp": {
        "gte": "1577983263"
      }
    }
  }
}

GET test-date/_search
{
  "size": 10, 
  "query": {
    "range": {
      "@timestamp": {
        "gte": "2020-01-01"
      }
    }
  }
}

I would suggest that you adjust the mapping of your index to reflect the actual formats used in the data. You can reindex from the already existing index into a new index that has epoch_millis in the date format (default).

1 Like
3

Awesome Magnus!
Your answer helped us a lot.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.