_update_by_query can't fill subfields in some documents

After dinamically updating index mapping (I added new subfield with a lowercase normaliizer), I ran an _update_by_query task to fill subfield with normalized values. However the task doesn't create subfield for several documents.

I added subfield "useLowercase" for field "canonical":

{
  "my_index" : {
    "mappings" : {
      "properties" : {
        "_class" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "canonical" : {
          "type" : "keyword",
          "fields" : {
            "keywordLowercase" : {
              "type" : "keyword",
              "ignore_above" : 256,
              "normalizer" : "lower_case_normalizer"
            },
            "useLowercase" : {
              "type" : "keyword",
              "ignore_above" : 256,
              "normalizer" : "useLowercase"
            }
          }
        },
        "certificationList" : {
          "properties" : {
....

Normalizer:

      "analysis" : {
          "normalizer" : {
...
            "useLowercase" : {
              "filter" : [
                "lowercase"
              ],
              "type" : "custom"
            }
          },
...

After that i ran _update_by_query task:

POST my_index/_update_by_query?slices=auto&conflicts=proceed&wait_for_completion=false
{
  "query": {
    "bool": {
      "must_not": [
        {
          "exists": {
            "field": "canonical.useLowercase"
          }
        }
      ]
    }
  }
}

The task updated almost all documents and was completed with no failures. However it left about 4.5k documents with empty canonical.useLowercase field. Query:

GET my_index/_search?track_total_hits=true
{
  "query": {
    "bool": {
      "must_not": [
        {
          "exists": {
            "field": "canonical.useLowercase"
          }
        }
      ],
      "must": [
        {
          "exists": {
            "field": "canonical"
          }
        }
      ]
    }
  }
} 

Starting _update_by_query one more time gave no result.

Question: why does it happen and how to update the rest of the documents?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.