Clarification on using date ranges for URI searches

[Jason_Baumgartner]

I am currently experimenting with using a Raspberry Pi 4 to ingest Reddit data in real-time. It is working very well, but I'm having a hard time finding concise documentation on how to use date ranges for a URI search.

For example, to set the range for the last hour I can do this:

http://pi4.pushshift.io/rc/_search/?pretty&sort=created_utc:desc&size=250&q=created_utc:[now-1h%20TO%20now]

This works as expected. However, if I try to use actual dates, I don't get back any data. For example:

http://pi4.pushshift.io/rc/_search/?pretty&sort=created_utc:desc&size=250&q=created_utc:["2019-01-01"%20TO%20"2020-01-01"]

I also tried including the time portion but that returned an empty set as well:

http://pi4.pushshift.io/rc/_search/?pretty&sort=created_utc:desc&size=250&q=created_utc:["2019-01-01T00:00:00"%20TO%20"2020-01-01T00:00:00"]

I would expect this to return results since I'm searching for a range for the entire year, but no documents are returned. What is the correct syntax when searching exact dates or datetime values?

The mapping for the created_utc field is as follows:

"created_utc": {
"type": "date",
"format": "epoch_second"
}

Also, this is the entire mapping for the index: http://pi4.pushshift.io/_mapping?pretty

These are the indices: http://pi4.pushshift.io/_cat/indices

[dadoonet (David Pilato)]

My guess is that because you defined "format":"epoch_second" in your mapping, elasticsearch can only parse dates which are epoch_second and not any other text.

May be you should try with an array like:

"format": ["OTHER_FORMAT", "epoch_second"]

That's a guess. I did not try to reproduce yet.

[dadoonet (David Pilato)]

So this works:

DELETE test
PUT test
{
  "mappings": {
    "properties": {
      "date": {
        "type": "date",
        "format": "dd/MM/yyyy||epoch_second"
      }
    }
  }
}
PUT test/_doc/1
{
  "date": "26/12/1971"
}
GET test/_search?q=date:["01/12/1971" TO "01/01/1972"]

Note that you can't use "epoch_second||dd/MM/yyyy". Only the first format is applied.

[Jason_Baumgartner]

Thank you David! I appreciate the reply and really enjoy your responses in this forum. You do a great job helping others with their questions. From one developer to another, you do a great service to the community.

My thought process in this example was that Elasticsearch would be able to internally translate the datetime to the appropriate epoch value.