# elasticsearch updateByQuery is not working for large data

**URL:** <https://discuss.elastic.co/t/elasticsearch-updatebyquery-is-not-working-for-large-data/363485>\
**Category:** Elasticsearch\
**Tags:** painless\
**Created:** [July 20, 2024, 8:26pm UTC](https://discuss.elastic.co/t/elasticsearch-updatebyquery-is-not-working-for-large-data/363485 "2024-07-20T20:26:15Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Karan\_Rawat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karan_rawat/32/136184_2.png) [@Karan\_Rawat](https://discuss.elastic.co/u/Karan_Rawat)\
**Post date:** [July 20, 2024, 8:26pm UTC](https://discuss.elastic.co/t/elasticsearch-updatebyquery-is-not-working-for-large-data/363485/1 "2024-07-20T20:26:15Z")

</div>

i have written a script that will create a key "source" on 4 indices of elasticsearch based on some condition . All of my 4 indices contain at least 2~3 million data . when i ran the script it added the key for only 3 entities , then i ran it again for the last entity (deals) exclusively. At last i noticed that some significant amount of docs were not having the new key which i introduced while running script . I can confirm that the conditions written in the script were correct . Here is the script code given below

```auto
const path = require('path');
require('dotenv').config({
    path: path.resolve(__dirname, '../.env')
});
const client = require('../server/utils/es-utils').initializeESConnection();

(async function addField(){

    const sq = {
        script: {
            source: `
                if (ctx._source.sourceCreated != null) {
                    if (ctx._source.sourceCreated.recordSource != null) {
                        ctx._source.sourceCreated.source = ctx._source.sourceCreated.recordSource;
                    } else if (ctx._source.sourceCreated.clientType != null) {
                        ctx._source.sourceCreated.source = ctx._source.sourceCreated.clientType.toLowerCase();
                    }
                }
            `,
            lang: "painless"
        }
    };

        const indices = ['contacts','accounts','deals','crm-activities'];
        for (const index of indices) {
            const response = await client.updateByQuery({
                index: index,
                body: sq,
                wait_for_completion: false,
                refresh: true
            });
            console.log(`Updating documents in index ${index}`);
        }
})()

```

I need to make sure that after running the script , i shall be able to see the key "source" on all of my docs of the given 4 indices

---

<div class="post-metadata">

**Author:** ![RabBit\_BR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rabbit_br/32/82261_2.png) [@RabBit\_BR](https://discuss.elastic.co/u/RabBit_BR)\
**Post date:** [July 21, 2024, 1:15pm UTC](https://discuss.elastic.co/t/elasticsearch-updatebyquery-is-not-working-for-large-data/363485/2 "2024-07-21T13:15:40Z")

</div>

@Karan_Rawat

You use update by query asynchronously, so a tip for your code would be to use the task ID for monitoring the operation. This way, you will know if any documents have conflicts and how many documents were updated.

---

<div class="post-metadata">

**Author:** ![Karan\_Rawat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karan_rawat/32/136184_2.png) [@Karan\_Rawat](https://discuss.elastic.co/u/Karan_Rawat)\
**Post date:** [July 21, 2024, 4:40pm UTC](https://discuss.elastic.co/t/elasticsearch-updatebyquery-is-not-working-for-large-data/363485/3 "2024-07-21T16:40:27Z")

</div>

I was constantly monitoring the tasks from Kibana. No such conflicts came up. One thing I do remember is that when I ran the query without the flag 'wait\_for\_completion: false', I received a query timeout. Did that lead to some inconsistency?

---

<div class="post-metadata">

**Author:** ![RabBit\_BR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rabbit_br/32/82261_2.png) [@RabBit\_BR](https://discuss.elastic.co/u/RabBit_BR)\
**Post date:** [July 22, 2024, 11:21am UTC](https://discuss.elastic.co/t/elasticsearch-updatebyquery-is-not-working-for-large-data/363485/4 "2024-07-22T11:21:52Z")

</div>

I can't confirm but maybe yes. In any case, before running the update you could validate how many documents would be updated, so, after applying the update you could compare the updated documents as expected. This would give you security.