Multiple Scroll requests concurrently on the same index drastically increase time for each scroll request compared to running individually

Elastic Stack Elasticsearch
1

We have a series of automated tasks that update different ES indexes with data. One piece of data is the people information that we also store in ES. What we are doing currently is using a scroll request to read in all the person information that is needed for that given task as an in memory cache. Sometimes all of these tasks run at the same time because of a server restart, or an upgrade to the task harness that manages all the tasks.

What we are seeing is that multiple scroll requests against the same index seem to go significantly slower. A single scroll request would normally be about a minute when no others are running. If that same request was then ran at the same time as other similar requests (somewhere around 3-6 additional requests) against that index it takes closer to 11 minutes for each to fully complete. Marvel hasn't shown that we are maxing out our resources on the ES server. I'm trying to figure out if there is something other than a scroll request that should be used to accomplish this, or a more efficient way to get the data we need. Keep in mind we are still on 1.x for ES as we haven't been able to move to the latest version of ES. So, if there are improvements around this in the later version then please include that in your answer as it would help me convince the team to upgrade sooner rather than later. The amount of data is roughly 2.4 million documents in the index.

2

Since you are on 1.x you should ensure that you combine the scroll with the scan search type.
The scan search type disabled the scoring and the sorting of the results so if the order doesn’t matter in your use case you should try this:
https://www.elastic.co/guide/en/elasticsearch/reference/1.7/search-request-scroll.html#scroll-scan

If you are already using the scan search type then I'd check the performance with two concurrent request, then three, ... You should see that the performance drops at some point but two or three concurrent requests should be ok.

3

Thanks for the response. We are using search_type=scan in our initial scroll request. 2 or 3 starts to slow down. Not necessarily up the the 11 minute, but starts getting up to about 5 minutes each. We are trying to find alternatives, like having a shared cache that we update to minimize this current slowdown. I mostly wanted to make sure we were using the scroll request correctly, and that there weren't other improvements within 2.x that would be useful to know about.