updateByQuery cluster state red

Elastic Stack Elasticsearch
1

Steps to reproduce:

  1. insert 45 000 000 documents
  2. run:
    let res = await client.updateByQuery({ index: config.get('index'), type: config.get('type'), slices: 5, waitForCompletion: false, conflicts: "proceed", requestTimeout: 36000000, timeout: -1, body: { "query": { "bool": { "must": { "match_all": {} }, "filter": { "bool": { "must": [ { "exists": { "field": "offers" } } ] } } } }, "script": { "lang": "painless", "inline": 'for (int i = 0; i < ctx._source.offers.length; i++) {ctx._source.offers[i].price_usd = ctx._source.offers[i].price / params.toUSD[ctx._source.offers[i].code]}', "params": { "toUSD": params } } } })
  3. After a while the cluster status will become red

Is it possible to update all records in index?

2

Why does the cluster go red? Are there any errors or warnings in the Elasticsearch logs? What does your documents look like?

3

We've just found the problem on one of our nodes.
No space left on device
During the updating elasticsearch eat a lot of space on disk. May be you know what can we do with it?

4

As Elasticsearch is working on of a point-in-time snapshot when the update by query operation starts, this dataset needs to remain, even if every document gets replaced. So if your update query operation affects a lot of documents, you need to have in the worst case twice the amount of index size available on disk.