Maximum Scroll Context

Elastic Stack Elasticsearch
1

Hi guys, I was wondering that the maximum scroll context is a node level property across all the indices right? If that’s true, then does it mean that any elastic cluster wherein let’s assume number of primary shards = number of data nodes, with each primary shard on 1 data node, the cluster cannot handle more than 1024 concurrent / parallel scroll requests? (Considering we have set the limit as 1024).

Also, if the data only exists on the 5 nodes, will the scroll instantly close on the remaining nodes, or will it continue to stay open till the time the entire polling is completed from these 5 nodes as well?

2

Welcome back @Lakshya_Gupta

It might be useful to reference the long previous discussion around your cluster, though I doubt anyone will read all of it (again):

Note I do recognize it's a different question(s() yu are asking today.

"if the data only exists on the 5 nodes, will the scroll instantly close on the remaining"

I'm not an expert on the internals, but I would expect no scroll context to be created on nodes with no matching data, never mind "instantly close"?

btw you can see the counts in output of a GET to

_nodes/stats/indices/search

so should not be too hard to validate the behavior

1 Like
3

That’s what I expected to happen, but apparently I created 2 scenarios.

Routing with routing partition = 5, and no routing.

In case of routing, each scroll request (containing the routing id) was only opening 5 scrolls, whereas in case of no routing, the scroll request was opening scrolls on all the primary shards.

Also just to mention, this is not associated with the other thread…this is a completely different :smiley: .

4

What does that mean? Maybe share/show what you actually did to "apparently create 2 scenarios" ?

It's an entirely different cluster? Just coincidence that the scroll request with routing id targeted only 5 nodes, 5 being the magic number in the other thread? :thinking:

If so, understood, please accept my apology.

1 Like
5

Basically creating 2 indices and populating them with the same data, however one had routing enabled and the other didn’t have any explicit custom routing enabled.

5 is our routing_partition which we have defined in the configuration, and hence it’s the magic number in both the threads :wink:

6

OK. And:

What's confusing you here?

But always best to be clear and explicit For me at least, it was really far from obvious that:

meant