# Best approach to update huge # of documents (millions) single query

**URL:** <https://discuss.elastic.co/t/best-approach-to-update-huge-of-documents-millions-single-query/345080>\
**Category:** Elasticsearch\
**Created:** [October 16, 2023, 8:59am UTC](https://discuss.elastic.co/t/best-approach-to-update-huge-of-documents-millions-single-query/345080 "2023-10-16T08:59:47Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![vtadmin](https://avatars.discourse-cdn.com/v4/letter/v/a9a28c/32.png) [@vtadmin](https://discuss.elastic.co/u/vtadmin)\
**Post date:** [October 16, 2023, 8:59am UTC](https://discuss.elastic.co/t/best-approach-to-update-huge-of-documents-millions-single-query/345080/1 "2023-10-16T08:59:47Z")

</div>

I'm using elastic for storing documents with multiple attributes that can go on and off (like active or inactive).  
They can go up to 1-2 million per index.

On a daily basis I need to synchronize those documents who can become OFF (or back ON, or insert new ones) with some data feeds.

Based on data feeds I know what IDs have status="on". Nothing else.  
My approach now is to preprocess and make a difference between all docs that had status="on" before synchronization, and after synch is finished.

Which gets me a list of ID's that will need to be updated with "off"  
Compared to total number those is usually 0.5-1%

I run 2 queries: A. update\_by\_query where I set status = "off" for docs in this list of ID's  
and B. one where I set status ="on" for docs not in the list of deleted ID's

```auto
{
  "query": {
    "bool": {
      "must_not": {
        "terms": {
          "_id": ["id1", "id2", "id3", ... , "idN"]
        }
      }
    }
  }
}

```

From time to time this list gets to a few hundred thousands.

My challenges:

- when this list gets really huge (once a month it happens), I can not make the query (it's MB in size)
- I can't split it in batches because I wouldn't know what to update

My question is how else could I do it?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 13, 2023, 8:59am UTC](https://discuss.elastic.co/t/best-approach-to-update-huge-of-documents-millions-single-query/345080/2 "2023-11-13T08:59:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.