I'm trying to pull results from elasticsearch with a sort. There are millions of records within each time period I'm looking at. It's taking a very long time to fetch all of the results. I'm looking for ways to improve the speed.
I looked at sliced scrolls, and it improves the time significantly, but the results are no longer really sorted. They are only sorted within their own slice, but I need the results to return in order. I tried using other fields to slice the scroll by, but all of the results end up with a lot of interleaving.
I saw this line in the docs for the sliced scroll:
Each scroll is independent and can be processed in parallel like any scroll request.
"Can be processed in parallel like any scroll request". Does that mean all scroll requests can be run in parallel? i.e. If I had a normal scroll and a single scroll id, could I fetch the results in order from multiple threads? (i.e. the first 10,000 results in one thread, then the second 10,000 in another, and so on)
Or does the line mean sliced scrolls can be processed in parallel, the same way as you would use any other scroll?
Maybe someone has another suggestion for how I could run a search query in parallel to get results faster on the client, but in order?
Thanks