After dinamically updating index mapping (I added new subfield with a lowercase normaliizer), I ran an _update_by_query task to fill subfield with normalized values. However the task doesn't create subfield for several documents.
I added subfield "useLowercase" for field "canonical":
{
"my_index" : {
"mappings" : {
"properties" : {
"_class" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"canonical" : {
"type" : "keyword",
"fields" : {
"keywordLowercase" : {
"type" : "keyword",
"ignore_above" : 256,
"normalizer" : "lower_case_normalizer"
},
"useLowercase" : {
"type" : "keyword",
"ignore_above" : 256,
"normalizer" : "useLowercase"
}
}
},
"certificationList" : {
"properties" : {
....
Normalizer:
"analysis" : {
"normalizer" : {
...
"useLowercase" : {
"filter" : [
"lowercase"
],
"type" : "custom"
}
},
...
After that i ran _update_by_query task:
POST my_index/_update_by_query?slices=auto&conflicts=proceed&wait_for_completion=false
{
"query": {
"bool": {
"must_not": [
{
"exists": {
"field": "canonical.useLowercase"
}
}
]
}
}
}
The task updated almost all documents and was completed with no failures. However it left about 4.5k documents with empty canonical.useLowercase field. Query:
GET my_index/_search?track_total_hits=true
{
"query": {
"bool": {
"must_not": [
{
"exists": {
"field": "canonical.useLowercase"
}
}
],
"must": [
{
"exists": {
"field": "canonical"
}
}
]
}
}
}
Starting _update_by_query one more time gave no result.
Question: why does it happen and how to update the rest of the documents?