I am getting below exception while running the update_by_query. We have been issuing this request to through Rest API (from our code) to OpensearchCluster in AWS.
Initially it seems ok, but when we have a sudden burst of request somewhere around 21k, we started getting this exception.
We have 2 indices and We are using 5 shards and 2 replica per index in OpenSearch.
org.opensearch.common.util.concurrent.OpenSearchRejectedExecutionException: Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.
The sample query looks like below
POST /<index-name>/_update_by_query
{
"script": {
"inline":"""
ctx._source['customers.name'] = params.name;
""",
"lang": "painless",
"params":{
"name": "ankit gupta"
}
},
"query": {
"term": {
"customers.handle.keyword": "date_user01"
}
}
}
- Are we getting this exception because we overwhelmed the Opensearch with http requests? We can not use bulk api as an option, as we are updating documents in the indexes whenever there is an event occurred.
- Should we use scroll parameter in the term request?
- How should we handle it when we have such large number of events happening across the application? What are the best practices to solve such use cases?
)