I'm trying to loop through all documents in a certain index using pagination (size,from) once ES limits the total documents returned to 10k, then this is my initial search (what I increase by 10k the from parameter to return the next 10k) :
Problem is that it doesn't allow me to start from > 10k meaning it's not a real pagination where i could start from 240000 and get the next 10k. The error message:
"reason": "Result window is too large, from + size must be less than or equal to:  but was . See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level parameter."
I also tried to use scan/scroll where it will search and bring all the results and I would need to loop through the _scroll_id and gather the sections of "size" (size*shards) to have all the documents. This is not a good option because:
- will exhaust the memory limits.
- Even if I decide to go for it, program stops (I believe because of the 10k limit again).
Thank you and hope somebody know if this is a issue or if this "pagination" should really work only for the first 10k docs.