Getting "too many scroll contexts" when re-indexing

[malsioufi]

Hi,

I have a task where I need to sort by the field "_id" I knew that I should not use this field for sorting and that instead it is better to add an extra field the has the same value as _id and then I can use that field for sorting. In order to achieve that I am trying to reindex my data.

POST _reindex?refresh
{
  "source": {
    "index": "my_original_index"
  },
  "dest": {
    "index": "my_new_index"
  },
  "script": {
    "source": "ctx._source.document_id = ctx._id"
  }
}

However, when I try to run the previous code I got the following error:
Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.

[spinscale (Alexander Reelsen)]

Are you using scroll searches in your regular operations already? Or is that index consisting of a lot of shards?

[malsioufi]

The index has 5 shards.
The index is used as a data source for a React application that is built with Reactivesearch and I think they do use scroll searches to get the data.

[spinscale (Alexander Reelsen)]

then there are probably a lot of scroll searches going on in parallel. Scroll searches are not supposed to be used by applications with many parallel queries.

In order to fix this, you could swithc those queries to use a single PointInTime reader and then update that one regularly, like every minute or thirty seconds. See Point in time API | Elasticsearch Guide [7.13] | Elastic