We are using ES version 6.4 and we want to upgrade to 7.4. While testing the up-gradation to do a paginated search using Scroll we are facing the below issue:
Trying to create too many scroll contexts. Must be less than or equal to [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.
Looks like we can open only 500 scroll contexts.
When we initiate a scroll, a scroll context (context A) is created and the scroll id points to the context (context A).
When I retrieve the next batch, a new scroll context (context B) is created. I am receiving the same scroll Id (now it points to context B).
What happens to the older scroll context(context A). Would it be closed and GC'd immediately? Or will happen only after the scroll time expires? How can I clear the older scroll context? The scroll Id associated with the older context (context A) is now pointing to the new one (context B).
How do we work around this issue?
Our options:
- We can increase scroll context in the settings. Are there any issues if we increase scroll contexts to a higher number (Ex: 10000)
- Clear the scroll contexts once it is used. We observed the same scroll id being returned to retrieve the subsequent batches. How do we clear the scroll context that was created for the older batch?