[Solved] Elasticsearch 7.2.0, Python and Date Range Query

[pritchey]

I'm really pulling my hair out on something that should be working (I think) but is returning no results. I'm using packetbeats to feed data into my Elasticsearch setup (7.2.0). Data is there - I can see it in Kibana.

I'm writing a Python (v3.7.4) client that can connect to my Elasticsearch server and it will pull data if I search a field for a specific value. The specific problem I'm having is with date ranges. Using Kibana, I can execute the following in the Console and it finds records (or a count):

GET packetbeat*/_count
{
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "@timestamp": {
              "gte": "now-1d",
              "lte": "now"
            }
          }
        }
      ]
    }
  }
}

If I take the above and "flatten" it in my Python code, no records are found. Here's the relevant snippet from my Python script:

   my_results = es.search(
        index="packetbeat*", 
        scroll='10m', 
        body={"query": {"bool":{"must":[{"range": {"@timestamp": {"gte": "now-1d","lte": "now"}}}]}}}, 
        size=20000
        )

What is it that I'm doing wrong? While I'm working on my script on a laptop and not my server, I've even copied the script over to the server and executed it there with the same exact results (no matching records).

(I have Googled more than I can shake a stick and tried many variations. My assumption is the query built in Kibana should work....since it works....)

[pritchey]

Found my problem!!!

Not giving up, I kept poking at things and finally decided to dump the entire contents of the search results instead of diving straight into (what should have been) results. That revealed this nugget:

{'type': 'illegal_argument_exception', 'reason': 'Batch size is too large, size must be less than or equal to: [10000] but was [20000]. Scroll batch sizes cost as much memory as result windows so they are controlled by the [index.max_result_window] index level setting.'}

Altering my size parameter (which had been working before I started playing with date ranges for some reason) and I now get search results.

[gabriel_tessier]

Hi @pritchey,

To retrieving big size data usually I use the helper scan, maybe more easy to manage:
https://elasticsearch-py.readthedocs.io/en/master/helpers.html?highlight=scroll#scan