Stored fields getting deleted upon partial update of the document in elastic search

Elastic Stack Elasticsearch
1

Hi.
I have an index which have stored fields in the documents. But, upon updating the document with new fields (partially update), the previously existing stored fields are getting deleted.

  1. Create the Index
PUT itf_test_index
{
  "mappings": {
    "_source": {
        "excludes": [ "inference*", "metric*"]
      },
    "dynamic_templates": [
      {
          "inference_fields": {
            "match": "inference*",
            "mapping": {
              "index": false,
              "store": true,
              "type": "text"
            }
          }
        },
        {
          "metric_fields": {
            "match": "metric*",
            "mapping": {
              "index": false,
              "store": true,
              "type": "text"
            }
          }
        }
    ]
  }
}
  1. Create a document
POST itf_test_index/_create/1
{
  "imageName": "123.jpg",
  "attribute_1": "Cat",
  "inference_1": "inference_1"
}
  1. Result of _search at this point
GET itf_test_index/_search
{
  "_source": ["*"],
  "stored_fields": ["*"]
}

######

"hits": [
      {
        "_index": "itf_test_index",
        "_id": "1",
        "_score": 1,
        "_source": {
          "imageName": "123.jpg",
          "attribute_1": "Cat"
        },
        "fields": {
          "inference_1": [
            "inference_1"
          ]
        }
      }
    ]
  1. Updating the document
POST itf_test_index/_update/1
{
  "doc": {
    "metric_1" : "metric_1"
  }
}
  1. Result at this point
GET itf_test_index/_search
{
  "_source": ["*"],
  "stored_fields": ["*"]
}

#######

"hits": [
      {
        "_index": "itf_test_index",
        "_id": "1",
        "_score": 1,
        "_source": {
          "imageName": "123.jpg",
          "attribute_1": "Cat"
        },
        "fields": {
          "metric_1": [
            "metric_1"
          ]
        }
      }
    ]

Here "inference_1" is not available any more. What exactly should I do at step 4 to preserve existing stored fields?

Thank you.

2

I am using Elastic search v8.9.0

3

Hi @Jagadeesh12

If you want to preserve the stored fields during updates, you will need to reindex the entire document by using the _index API. This involves fetching the entire document, making the necessary modifications (including the new field), and then reindexing the updated document.

Here's how you can achieve that:

  1. Retrieve the existing document using the _source field.
    ex: GET itf_test_index/_source/1
  2. Modify the retrieved document by adding the new field(s) you want to update along with the existing fields you want to keep.
    ex: { "imageName": "123.jpg", "attribute_1": "Cat", "inference_1": "inference_1", "metric_1": "metric_1" }
  3. Perform the _index request with the modified document to reindex it.
PUT itf_test_index/_doc/1   
{
  "imageName": "123.jpg",
  "attribute_1": "Cat",
  "inference_1": "inference_1",
  "metric_1": "metric_1"
}
4

Hi @DineshNaik , Thanks for the reply.
I get that I can re-index by fetching the entire source and stored fields, but that's exactly what I am trying to avoid. Elastic search allows partial update on _source (where it clubs new fields with existing _source), but how come it doesn't allow the same on stored fields?

Why does it have to delete the existing stored fields?

5

If your primary concern is to avoid re-fetching and reindexing the entire document, you might consider using the _source field instead of stored fields for fields that require partial updates.
However, this might increase the storage requirements, so it's crucial to carefully evaluate your use case and the trade-offs involved.

When you perform a partial update on the _source field, Elasticsearch can efficiently merge the new fields with the existing _source JSON. However, when it comes to stored fields (stored in doc_values ), there is no efficient way to do a partial update and merge them with the existing stored fields.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.