# Bulk Update operation on multi field type and search

**URL:** <https://discuss.elastic.co/t/bulk-update-operation-on-multi-field-type-and-search/319143>\
**Category:** Elasticsearch\
**Created:** [November 17, 2022, 5:47am UTC](https://discuss.elastic.co/t/bulk-update-operation-on-multi-field-type-and-search/319143 "2022-11-17T05:47:18Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sagar\_Viradiya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sagar_viradiya/32/113024_2.png) [@Sagar\_Viradiya](https://discuss.elastic.co/u/Sagar_Viradiya)\
**Post date:** [November 17, 2022, 5:47am UTC](https://discuss.elastic.co/t/bulk-update-operation-on-multi-field-type-and-search/319143/1 "2022-11-17T05:47:18Z")

</div>

Previously we had mapping in `index_1` Index.

```auto
{
    "properties": {
       "offer_id":{
        "type":long,
         "fields": {
                "keyword": {
                    "type": "text"
                }
            }
},
        "seller_account_id": {
            "type": "long"
          
        }
    }
}

```

and all data are there.  
We were not able to search `seller_account_id` when provided in multi\_match for full text search  
So we decided to update mapping for `seller_account_id` as multi\_type field.

```auto
{
    "properties": {
        "seller_account_id": {
            "type": "long",
            "fields": {
                "keyword": {
                    "type": "text"
                }
            }
        }
    }
}

```

**I believe no Re-indexing is required for update mapping, but how we should make data available in search, because when we updated mapping search was not working?, we tried to use bulk API to update all documents. but still old documents are not searchable , only new documents are coming in search, I tried with \_bulk API with \_refresh=true, but still no luck**

Do I need to create new Index and reindex all documents.( I dont prefer this)  
Our multi\_match request

```auto
{
    "query": {
         "multi_match": {
             "fields": [
                 "offer_id.keyword",
                 "seller_account_id.keyword"
             ],
             "operator": "and",
             "query": "9333 8029425",
             "type": "cross_fields"
         }
    }
}   

```

Is not working.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 17, 2022, 6:55am UTC](https://discuss.elastic.co/t/bulk-update-operation-on-multi-field-type-and-search/319143/2 "2022-11-17T06:55:46Z")

</div>

Have you tried using the [update by query API](https://www.elastic.co/guide/en/elasticsearch/reference/8.5/docs-update-by-query.html) to process old documents so the new mapping takes effect?

---

<div class="post-metadata">

**Author:** ![Sagar\_Viradiya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sagar_viradiya/32/113024_2.png) [@Sagar\_Viradiya](https://discuss.elastic.co/u/Sagar_Viradiya)\
**Post date:** [November 17, 2022, 6:59am UTC](https://discuss.elastic.co/t/bulk-update-operation-on-multi-field-type-and-search/319143/3 "2022-11-17T06:59:30Z")

</div>

I was just checking examples from that document

```auto
POST test/_update_by_query?refresh&conflicts=proceed
POST test/_search?filter_path=hits.total
{
  "query": {
    "match": {
      "flag": "foo"
    }
  }
}

```

Are both different request or same?  
Do I just need to use `POST test/_update_by_query?refresh&conflicts=proceed` without body?

Actually I did not understand example, If you can help me on it.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 17, 2022, 7:05am UTC](https://discuss.elastic.co/t/bulk-update-operation-on-multi-field-type-and-search/319143/4 "2022-11-17T07:05:37Z")

</div>

I would identify a few old documents that need to be updated and run an update by query with a filter to target just a few documents in order to verify that it works and resolves the issue. Once that is done you should be able to run the task without a body and process all documents. To be more selective you might also be able to write a query clause to select only documents that do not have the `seller_account_id.keyword` field defined.

---

<div class="post-metadata">

**Author:** ![Sagar\_Viradiya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sagar_viradiya/32/113024_2.png) [@Sagar\_Viradiya](https://discuss.elastic.co/u/Sagar_Viradiya)\
**Post date:** [November 17, 2022, 7:22am UTC](https://discuss.elastic.co/t/bulk-update-operation-on-multi-field-type-and-search/319143/5 "2022-11-17T07:22:37Z")

</div>

How can I check if there is `seller_account_id.keyword` field is defined or not? can you share example.  
Because if mapping is updated all document have `seller_account_id` available with `seller_account_id.keyword` right?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 17, 2022, 7:28am UTC](https://discuss.elastic.co/t/bulk-update-operation-on-multi-field-type-and-search/319143/6 "2022-11-17T07:28:58Z")

</div>

> [@Sagar\_Viradiya](#):
>
> How can I check if there is `seller_account_id.keyword` field is defined or not?

You can add a boolean query with a must\_not [exists clause](https://www.elastic.co/guide/en/elasticsearch/reference/8.5/query-dsl-exists-query.html).

> [@Sagar\_Viradiya](#):
>
> Because if mapping is updated all document have `seller_account_id` available with `seller_account_id.keyword` right?

Elasticsearch stores data in immutable segments. The new field will therefore not be added for existing documents unless you update them, causing them to be written to a new segment.

---

<div class="post-metadata">

**Author:** ![Sagar\_Viradiya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sagar_viradiya/32/113024_2.png) [@Sagar\_Viradiya](https://discuss.elastic.co/u/Sagar_Viradiya)\
**Post date:** [November 17, 2022, 8:50am UTC](https://discuss.elastic.co/t/bulk-update-operation-on-multi-field-type-and-search/319143/7 "2022-11-17T08:50:53Z")

</div>

So there will not be required `_reindex` API to use in overall solution? Generally what steps should be follow when updating field as `multi_type` field in existing index? and also existing data should be available for searching?

If we use `update_by_query` , how we can resolve conflicts in existing documents?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 17, 2022, 8:58am UTC](https://discuss.elastic.co/t/bulk-update-operation-on-multi-field-type-and-search/319143/8 "2022-11-17T08:58:13Z")

</div>

> [@Sagar\_Viradiya](#):
>
> So there will not be required `_reindex` API to use in overall solution?

I believe that is correct.

> [@Sagar\_Viradiya](#):
>
> and also existing data should be available for searching?

Yes. Run a refresh before starting the update as the query clause relies on all old data being searchable.

> [@Sagar\_Viradiya](#):
>
> If we use `update_by_query` , how we can resolve conflicts in existing documents?

If you have conflicts, I believe you should skip those documents as it indicates that a new version was indexed after you started the operation and that means it will automatically get the new mappings.

---

<div class="post-metadata">

**Author:** ![Sagar\_Viradiya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sagar_viradiya/32/113024_2.png) [@Sagar\_Viradiya](https://discuss.elastic.co/u/Sagar_Viradiya)\
**Post date:** [November 17, 2022, 8:58am UTC](https://discuss.elastic.co/t/bulk-update-operation-on-multi-field-type-and-search/319143/9 "2022-11-17T08:58:26Z")

</div>

When I used

```auto
{
    "query": {
        "bool": {
            "must_not": {
                "exists": {
                    "field": "seller_account_id.keyword"
                }
            }
        }
    }
}

```

It returns only results where `seller_account_id` is null and there are few. but I have issue in many records.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 17, 2022, 8:59am UTC](https://discuss.elastic.co/t/bulk-update-operation-on-multi-field-type-and-search/319143/10 "2022-11-17T08:59:12Z")

</div>

Then you may need to update all documents.

---

<div class="post-metadata">

**Author:** ![Sagar\_Viradiya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sagar_viradiya/32/113024_2.png) [@Sagar\_Viradiya](https://discuss.elastic.co/u/Sagar_Viradiya)\
**Post date:** [November 17, 2022, 9:01am UTC](https://discuss.elastic.co/t/bulk-update-operation-on-multi-field-type-and-search/319143/11 "2022-11-17T09:01:51Z")

</div>

I am thinking another solution

like adding temporary field like `temp_id` and bulk update all documents with this field , so document version would be updated and it will be available for search, right?

And after that I will remove field by updating mapping.

Let me know your thoughts

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 17, 2022, 9:03am UTC](https://discuss.elastic.co/t/bulk-update-operation-on-multi-field-type-and-search/319143/12 "2022-11-17T09:03:31Z")

</div>

I would just run a update by query on all documents and skip on conflicts.

Adding a field requires an update to every document and you can not remove fields from mappings without reindexing. Sounds a lot more complicated...

---

<div class="post-metadata">

**Author:** ![Sagar\_Viradiya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sagar_viradiya/32/113024_2.png) [@Sagar\_Viradiya](https://discuss.elastic.co/u/Sagar_Viradiya)\
**Post date:** [November 17, 2022, 9:04am UTC](https://discuss.elastic.co/t/bulk-update-operation-on-multi-field-type-and-search/319143/13 "2022-11-17T09:04:48Z")

</div>

Ok, Thanks. running `update_by_query` for all documents mean there will be no critieria to search, I mean `update_by_query` would require some body right?

---

<div class="post-metadata">

**Author:** ![Sagar\_Viradiya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sagar_viradiya/32/113024_2.png) [@Sagar\_Viradiya](https://discuss.elastic.co/u/Sagar_Viradiya)\
**Post date:** [November 17, 2022, 9:10am UTC](https://discuss.elastic.co/t/bulk-update-operation-on-multi-field-type-and-search/319143/14 "2022-11-17T09:10:48Z")

</div>

Is it correct to update all documents?

```auto
POST test/_update_by_query?refresh&conflicts=proceed
{
  "query": {
    "match_all": {}
  }
}

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 17, 2022, 9:21am UTC](https://discuss.elastic.co/t/bulk-update-operation-on-multi-field-type-and-search/319143/15 "2022-11-17T09:21:09Z")

</div>

That should work but the first example in the docs I linked to seems to do exactly what you want.

---

<div class="post-metadata">

**Author:** ![Sagar\_Viradiya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sagar_viradiya/32/113024_2.png) [@Sagar\_Viradiya](https://discuss.elastic.co/u/Sagar_Viradiya)\
**Post date:** [November 17, 2022, 9:22am UTC](https://discuss.elastic.co/t/bulk-update-operation-on-multi-field-type-and-search/319143/16 "2022-11-17T09:22:26Z")

</div>

This one?

```auto
POST my-index-000001/_update_by_query?conflicts=proceed

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 17, 2022, 9:35am UTC](https://discuss.elastic.co/t/bulk-update-operation-on-multi-field-type-and-search/319143/17 "2022-11-17T09:35:14Z")

</div>

Yes.

---

<div class="post-metadata">

**Author:** ![Sunile\_Manjee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunile_manjee/32/111461_2.png) [@Sunile\_Manjee](https://discuss.elastic.co/u/Sunile_Manjee)\
**Post date:** [November 29, 2022, 5:51am UTC](https://discuss.elastic.co/t/bulk-update-operation-on-multi-field-type-and-search/319143/18 "2022-11-29T05:51:51Z")

</div>

Curious if you tried run time field? this would be avoid updates to existing docs.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 27, 2022, 5:52am UTC](https://discuss.elastic.co/t/bulk-update-operation-on-multi-field-type-and-search/319143/19 "2022-12-27T05:52:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.