# Update\_by\_query fails to update all documents in ElasticSearch, getting true in between

**URL:** <https://discuss.elastic.co/t/update-by-query-fails-to-update-all-documents-in-elasticsearch-getting-true-in-between/316744>\
**Category:** Elasticsearch\
**Tags:** painless\
**Created:** [October 17, 2022, 8:11am UTC](https://discuss.elastic.co/t/update-by-query-fails-to-update-all-documents-in-elasticsearch-getting-true-in-between/316744 "2022-10-17T08:11:32Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![PriyaSmartShore](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priyasmartshore/32/112138_2.png) [@PriyaSmartShore](https://discuss.elastic.co/u/PriyaSmartShore)\
**Post date:** [October 17, 2022, 8:11am UTC](https://discuss.elastic.co/t/update-by-query-fails-to-update-all-documents-in-elasticsearch-getting-true-in-between/316744/1 "2022-10-17T08:11:32Z")

</div>

I have over 25 million documents in Elasticsearch (version - 7.6.2), those are having duplicates among them. So I have written a script in painless to delete one instance of duplicated document. I have tried running the script on test index (having approx 50k records) for that it works fine, but when tested on the main index which is having 25 millions of documents data it usually only process 300k documents and then it stops.

Script/query is running in background using wait for completion also timeout has been mentioned as in days . Also there is cron running which is cleaning/freeing up RAM every 5 minutes and the server space which is FREE is approx 120 GB (which I thought of to be sufficient as the script is not writing anything it is deleting the records).

Query looks like

````auto
POST testing_index/_update_by_query?wait_for_completion=false && timeout=4d
{
   "script": {
    "source": """      
      String replace(String word) {
      
        def prefixUrl = 'file://abc.directory.intra/homes/test';
        
        if(word.contains('xyz')) {
          prefixUrl = 'file://xyz.directory.intra/homes/test';
        }
        
        String[] pieces = word.splitOnToken(prefixUrl);
        int lastElIndex = pieces.length - 2;
        pieces[lastElIndex] = '';
        def list = Arrays.asList(pieces);
        return String.join('',list);
      }
 
     
      def path = replace(ctx._source.d_path);
      def matchResult = 'no';
      
      for(int i=0;i<params.pathArray.length;i++) {
        
        if(params.pathArray[i] == path && path!="") {
          
          matchResult = 'yes';
          params.pathArray.remove(i);
        }
      }
      
      if(matchResult == 'yes') {
        ctx.op = 'delete';
        
      } else {
        ctx.op = 'noop';
        params.pathArray.add(path);
      }
      
    """,
    "lang": "painless",
    "params": {
      "pathArray": []
    }
  }
}
  
{
  "completed": true, --why getting true in between
  "task": {
    "node": "Jsecb8kBSdKLC47Q28O6Pg",
    "id": 5968304,
    "type": "transport",
    "action": "indices:data/write/update/byquery",
    "status": {
      "total": 24002005,
      "updated": 0,
      "created": 0,
      "deleted": 333567,
      "batches": 137,
      "version_conflicts": 0,
      "noops": 0,
      "retries": {
        "bulk": 0,
        "search": 0
      },
    
    }
```After processing almost 300k documents it turns out in "true" (same behavior has been seen 2,3 times of testing) and then not able to get why it is not able to process whole documents. When with small number of records it worked well

Reason I doubt is :- The maximum number of statements that can be executed in a loop has been reached

How can I increase this limit so that script can processed all 25 millions of documents in one go.`Preformatted text`
````

---

<div class="post-metadata">

**Author:** ![stu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stu/32/75063_2.png) [@stu](https://discuss.elastic.co/u/stu)\
**Post date:** [October 17, 2022, 1:55pm UTC](https://discuss.elastic.co/t/update-by-query-fails-to-update-all-documents-in-elasticsearch-getting-true-in-between/316744/2 "2022-10-17T13:55:27Z")

</div>

Hi, it seems that the space in use by `params` is indeed the issue, however using `params` as a way to pass state between invocations of the script is not supported.

Elasticsearch does not guarantee when new instances of `params` is materialized but currently that occurs every new segment.

Scripts should be stateless unless the context has been specifically designed to pass state, like [scripted metric aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-scripted-metric-aggregation.html).

A [reindex request](https://www.elastic.co/guide/en/elasticsearch/painless/current/painless-reindex-context.html) could change the `id` to a unique value, which is one way to get rid of duplicates, perhaps that would work in this case?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 14, 2022, 1:55pm UTC](https://discuss.elastic.co/t/update-by-query-fails-to-update-all-documents-in-elasticsearch-getting-true-in-between/316744/3 "2022-11-14T13:55:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.