ElasticSearch delete_by_query: Trying to create too many scroll contexts

[lankin1 (lankin)]

Hi,
I am trying to fix this error while running delete_by_query API on AWS ElasticSearch:

Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.

Changing the search.max_open_scroll_context cluster setting gives back:
{"Message":"Your request: '/_cluster/settings' payload is not allowed."}

I've found that it is possible to send a request for changing that setting to the AWS support team, but it is not my preferred solution since upping the limit might have some negative side affects.

I am using elasticsearch version 7.7. Are there any other solutions to this problem?

Thanks in advance

[HenningAndersen]

Hi @lankin1 ,

are you doing 100s of delete_by_query requests concurrently or just one at the time? Doing other workloads using scroll queries? Does the error happen immediately after you start the delete_by_query or after some time? Perhaps look at _nodes/stats to see if the number of scroll contexts increase over time.

The error when changing search.max_open_scroll_context sounds like a problem with your request to change it, can you paste the request you are trying to do here?

[dadoonet (David Pilato)]

@lankin1

AWS Elasticsearch:

Note that we don't support this service here.

BTW did you look at Cloud by Elastic, also available if needed from AWS Marketplace ?

Cloud by elastic is one way to have access to all features, all managed by us. Think about what is there yet like Security, Monitoring, Reporting, SQL, Canvas, Maps UI, Alerting and built-in solutions named Observability, Security, Enterprise Search and what is coming next ...