Getting many more results than expected

[Shlomo_Koppel]

Hi,
I am making the following query:
{'bool': {'must': [{'terms': {'doc.attributes.type.keyword': ['Attachment', 'Document']}}, {'terms': {'doc.internal_id': ['xxxx83a1c00f7004b52dxxxx']}}], 'filter': [{'range': {'doc.date_es_index_timestamp': {'lte': 1696569227789}}}]}}

I am expecting to get a single hit in the results since there is only 1 document with this id. Yet when I run it, it paginates over 100 pages until I get the error:

java.lang.IllegalArgumentException: Result window is too large, from + size must be less than or equal to: [10000] but was [10100]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.

The above search appears to be translated to:

{"from":10000,"size":100,"query":{"bool":{"must":[{"terms":{"doc.attributes.type.keyword":["Attachment","Document"],"boost":1.0}},{"terms":{"doc.internal_id":["xxxx83a1c00f7004b52dxxxx"],"boost":1.0}}],"filter":[{"range":{"doc.date_es_index_timestamp":{"from":null,"to":1696569227789,"include_lower":true,"include_upper":true,"boost":1.0}}}]

I am wondering why I would be getting such a large amount of results (over 10000).
Thank you

[dadoonet (David Pilato)]

Welcome!

How are you running the first query? From which tool or code?

[Shlomo_Koppel]

Thank you!
I am running using python library elasticsearch_dsl.
Our code calling elastic looks like the following:

       s = Search(using=self._client, index=self._index_pattern).query(query).source([f'doc.{field}' for field in source_fields])
        skip = (page - 1) * size if page > 0 else 0
        s = s[skip:skip + size]
        response = s.execute()

[dadoonet (David Pilato)]

I'm not sure about what you are doing.

Why not using from and size parameters?

https://elasticsearch-py.readthedocs.io/en/v8.10.1/api.html#elasticsearch.Elasticsearch.search