Max_open_scroll_context, scroll_ids, and shards

[Michael_Cayanan]

Hi. I'm making a query across several indices with 8 shards per index. To be specific it ends up being about 752 shards in total.

When I make a scroll query, I initially get that

Trying to create too many scroll contexts

error since I'm going over the 500 default limit. I'm making sure that we are deleting the scroll_ids every time we finish the scroll functionality. So I increase the max_open_scroll_context limit it to a larger number like 1500 and now the scroll query works fine and no longer errors out.

I guess my question is, is the number of shards equal to the number of max_open_scroll_context ? It seems like ElasticSearch needs to open as many scroll contexts as there is shards. If that is true, why is that the case?

[HenningAndersen]

Hi @Michael_Cayanan,

yes, it will need a search context per shard, but the limit is per node. I am guessing your setup is a single node setup? Using 8 shards per index, it sounds like you are planning to scale out the deployment? If not, you could likely benefit from having fewer shards per index.

Elasticsearch needs a search context per shard it searches. When using scrolls, Elasticsearch will in many cases need to keep those search contexts alive for the duration of the scroll. For non-scroll searches, the search context can be closed as soon as we know what the top hits of the query are.

[Michael_Cayanan]

Hi @HenningAndersen. Thanks for the explanation on how scrolling works in relation to shards. Regarding my setup, I believe we do plan on scaling out if need be. But just curious, if we maintained a single node setup, how many shards per index would you recommend?

[HenningAndersen]

1 primary shard (which is also default) is fine in most cases, in particular for single node setups. If you have multiple nodes and one index is more heavily ingested into or searched against, partitioning into multiple shards makes sense.