Negative key value for date_histogram in epoch_milli

Elastic Stack Elasticsearch
1

Hi ES Dev Team,

One of the breaking changes in version 7.0 documentation says that negative epoch timestamps are no longer supported.

In our application, we use negative epoch timestamps as a lot of our documents have dates earlier than 1970. We use the date_histogram aggregation on these dates, and the "key" field in the response from date_histogram aggregation is formatted in epoch_milli. The response keys still come in negative epoch timestamps. Does that mean it is still supported?

Will the key format be changed down the road as well? What would be your suggestion for us moving forward?

Thanks in advance,
Jenna

2

Hi can someone help me with this

3

Hey,

this statement IIRC refers to negative epoch timestamps when indexing documents and the field is configured as a type: date with a mapping of epoch_*. This is not about date histogram aggregation.

So, how do you index your data?

--Alex

4

Hi Alex,

Thanks for looking into this!

From my understanding, you are saying that negative epoch timestamps is only invalid in index time, but still remains supported and valid at query time. Is that correct?

Our mapping for the date looks something like:

PUT test-date/_mapping
{
"properties": {
"sort_date": {
"type": "date",
"format": "yyyy-MM-dd G || yyyy-MM-dd || yyyy-MM || Y G || Y || date_time_no_millis || date_hour_minute_second || epoch_millis"
}
}
}

And in this case, we have run into another issue with epoch_millis. When using the range query, the epoch_millis format does not recognize any epoch_millis values with its last 3 digits being anything other than '000'.

To replicate this, using the mapping posted above and put these docs:

PUT test-date/_doc/1
{
  "sort_date": "0001-12-31 AD"
}
PUT test-date/_doc/2
{
  "sort_date": "0005-03-12 AD"
}
PUT test-date/_doc/3
{
  "sort_date": "0005-03-12 AD"
}

Then run the following query:

GET test-date/_search
{
"query": {
"bool": {
"must": {
"query_string": {
"default_operator": "AND",
"query": "*"
}
},
"filter": {
"bool": {
"must": [
{
"range": {
"sort_date": {
"gte": "-62559940430768",
"lte": "-59492208738460",
"format": "epoch_millis"
}
}
}
]
}
}
}
}
}

We then receive an error saying
failed to parse date field [-62559940430768] with format [epoch_millis]: [failed to parse date field [-62559940430768] with format [epoch_millis]]

However, if I change the last 3 digits of the above numbers to 000, the query works and return results successfully.

Any ideas or suggestions?

Thanks in advance!
Jenna

5

ah, now I understand your use case, thanks for providing the examples.

Judging from https://github.com/elastic/elasticsearch/blob/master/server/src/main/java/org/elasticsearch/common/time/EpochTime.java the parsing of negative epoch_millis timestamp should fail. As only positive numbers are allowed.

Can you provide a reproducible example including indexing/mapping to dig deeper?

6

Hi Alex,

I have included a simplified mapping and reproducible docs and queries in my previous response above. Is there anything else I can provide you with?

Thanks,
Jenna

7

Thanks for updating the snippet:

"range": {
  "sort_date": {
    "gte": "-62559940430768",
    "lte": "-59492208738460",
    "format": "epoch_millis"
  }
}

this is a negative epoch date and thus the exception is thrown. Replace this with a UTC8601 formatted date...

--Alex

8

However this query returns successfully if I change the last 3 digits of the timestamps to 000:

"range": {
  "sort_date": {
    "gte": "-62559940430000",
    "lte": "-59492208738000",
    "format": "epoch_millis"
  }
}

Any reason for this inconsistency?

Thanks,
Jenna

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.