Query @timestamp return epoch time

Elastic Stack Elasticsearch
1

When i query elasticsearch the @timestamp field comes back at iso8601 format. is there a way to get the field back as epoch time instead? the @timestamp field in my index is the regular 'date' field for elastic search.

everything works fine for querying with date ranges and such, but i'd like to work on the data in a program that uses epoch time instead of iso8601.

i can convert iso8601 to epoch time in perl, but it's way slow using the DateTime module. it would be nicer if elasticsearch would just return the epoch time in the field instead of iso8601

1 Like
2

Interestingly, Elasticsearch stores its dates as epoch milliseconds unless you explicitly disable doc_values for that field, so it's not too difficult to fetch it as such.

Two suggestions:

  1. index both forms of the date (e.g. "date_iso8601" and "date_epoch") so that you can use whichever is most convenient for a given operation
  2. pull the underlying docvalue value:
PUT my_index
{
  "mappings": {
    "my_type": {
      "properties": {
        "date": {
          "type":   "date",
          "format": "date_time_no_millis"
        }
      }
    }
  }
}

PUT my_index/my_type/1
{
  "date": "2017-01-01T12:00:00-05:00" 
} 

GET my_index/_search 
{
  "query":{
    "match_all": {}
  }, 
  "docvalue_fields": ["date"]
}
1 Like
3

Thanks. I ended up just putting the time in both formats into the index, one as a long and the other as date. i'll look at docvalue_fields for the next project. i was hopeful since the date was already in epoch time, i could just get the date in epoch or some other format as part of the display of the data rather then add more data to the index. but i image its more complicated under the covers then that.

4

That #2 option I proposed was indeed a way to get the underlying epoch time out without having to store two copies of the date. That said, it does seem a bit clunky and also doesn't work as part of the Document API (GET /{index}/{type}/{id}). Considering the optimizations around storing and retrieving longs, I think the extra flexibility gained with the duplication is probably worth it, but you'd be the best to judge!

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.