Not able to search or aggregate values from a field included lately in the mapping

[Fabio_Batalha]

Hello there!

I have a field (log_file_name) in my index that was not included in the mapping since de index was created. All records have the field but no mapping was defined to it. My mapping is configured as dynamic = False.

After a while, I realized it was important to allow search and aggregation using this field, so I updated the mapping to include it.

PUT index_name/_mapping
{
"properties" : {
"log_file_name" : {
"type" : "keyword",
"index": true
}
}

The mapping was updated since the configuration for this field is now available when I request the current mapping.

GET index_name/_mapping

Even though everything is well configured, I'm still not able to query or aggregate values of the field log_file_name.

As far as I now, it should be possible to change the mapping (to include a config for a pre-existing field) without the need to re-index all the index.

Thanks
Fabio

[DineshNaik (Dinesh Kumar Naik)]

@Fabio_Batalha
Which version of elastic is this?

Share the output of
GET index_name/_mapping

The keyword and text datatypes haven’t always been part of Elasticsearch. Originally, Elasticsearch provided just a single string datatype, and users could set an option called index to either analyzed or not_analyzed in their mapping to specify whether they wanted a string to be broken down into its individual terms upon indexing or simply indexed as-is.

Take a look at this post:

Elasticsearch replaces string type with two new types text and keyword.

[Fabio_Batalha]

Hello @DineshNaik

I noticed the index is working for the field log_file_name only for new records inserted after the mapping change.

My ES is:

{
  "name" : "es01.prod",
  "cluster_name" : "es-cluster",
  "cluster_uuid" : "86aqM3XBTVaTtrV5_-RrOg",
  "version" : {
    "number" : "7.6.2",
    "build_flavor" : "oss",
    "build_type" : "docker",
    "build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
    "build_date" : "2020-03-26T06:34:37.794943Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

The Mapping:

{
  "site_usage_v11_2019" : {
    "mappings" : {
      "dynamic" : "false",
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "access_method" : {
          "type" : "keyword"
        },
        "access_month" : {
          "type" : "keyword",
          "index" : false
        },
        "access_type" : {
          "type" : "keyword"
        },
        "access_year" : {
          "type" : "keyword",
          "index" : false
        },
        "accessed_uri" : {
          "type" : "text",
          "index" : false
        },
        "article_version" : {
          "type" : "keyword"
        },
        "audit" : {
          "properties" : {
            "identifier" : {
              "type" : "keyword"
            },
            "is_article_concentrated" : {
              "type" : "boolean"
            },
            "is_journal_concentrated" : {
              "type" : "boolean"
            },
            "is_legitimate" : {
              "type" : "boolean"
            }
          }
        },
        "document_treatment_type" : {
          "type" : "keyword"
        },
        "document_type" : {
          "type" : "keyword"
        },
        "doi" : {
          "type" : "keyword",
          "index" : false
        },
        "electronic_issn" : {
          "type" : "keyword",
          "index" : false
        },
        "geoip" : {
          "properties" : {
            "city_name" : {
              "type" : "keyword"
            },
            "continent_code" : {
              "type" : "keyword",
              "index" : false
            },
            "country_code2" : {
              "type" : "keyword"
            },
            "country_code3" : {
              "type" : "keyword"
            },
            "country_name" : {
              "type" : "keyword"
            },
            "dma_code" : {
              "type" : "long",
              "index" : false
            },
            "ip" : {
              "type" : "keyword"
            },
            "latitude" : {
              "type" : "float",
              "index" : false
            },
            "location" : {
              "type" : "geo_point",
              "index" : false
            },
            "longitude" : {
              "type" : "float",
              "index" : false
            },
            "postal_code" : {
              "type" : "keyword",
              "index" : false
            },
            "region_code" : {
              "type" : "keyword",
              "index" : false
            },
            "region_name" : {
              "type" : "keyword"
            },
            "timezone" : {
              "type" : "keyword",
              "index" : false
            }
          }
        },
        "http_code" : {
          "type" : "keyword",
          "index" : false
        },
        "http_method" : {
          "type" : "keyword",
          "index" : false
        },
        "http_status_code" : {
          "type" : "keyword",
          "index" : false
        },
        "institution_identifiers" : {
          "properties" : {
            "type" : {
              "type" : "keyword",
              "index" : false
            },
            "value" : {
              "type" : "keyword",
              "index" : false
            }
          }
        },
        "institution_name" : {
          "type" : "keyword",
          "index" : false
        },
        "institution_proprietary_id" : {
          "type" : "keyword"
        },
        "ip" : {
          "type" : "ip",
          "index" : false
        },
        "is_double_click" : {
          "type" : "boolean"
        },
        "is_request" : {
          "type" : "boolean"
        },
        "is_retro" : {
          "type" : "boolean"
        },
        "is_scholarly_journal" : {
          "type" : "boolean"
        },
        "is_subscribed_to_journal" : {
          "type" : "boolean"
        },
        "is_subscriber" : {
          "type" : "boolean"
        },
        "is_unique" : {
          "type" : "boolean"
        },
        "issue_id" : {
          "type" : "keyword"
        },
        "issue_label" : {
          "type" : "keyword",
          "index" : false
        },
        "issue_publication_year" : {
          "type" : "keyword"
        },
        "issue_serialization_year" : {
          "type" : "keyword"
        },
        "item_contributors" : {
          "properties" : {
            "identifier" : {
              "type" : "keyword",
              "index" : false
            },
            "name" : {
              "type" : "keyword",
              "index" : false
            },
            "type" : {
              "type" : "keyword",
              "index" : false
            }
          }
        },
        "item_format" : {
          "type" : "keyword"
        },
        "item_identifiers" : {
          "properties" : {
            "type" : {
              "type" : "keyword",
              "index" : false
            },
            "value" : {
              "type" : "keyword",
              "index" : false
            }
          }
        },
        "log_file_name" : {
          "type" : "keyword"
        },
        "log_type" : {
          "type" : "keyword"
        },
        "parent_data_type" : {
          "type" : "keyword"
        },
        "parent_doi" : {
          "type" : "keyword",
          "index" : false
        },
        "parent_item_contributors" : {
          "properties" : {
            "name" : {
              "type" : "keyword",
              "index" : false
            },
            "type" : {
              "type" : "keyword",
              "index" : false
            }
          }
        },
        "parent_item_identifiers" : {
          "properties" : {
            "type" : {
              "type" : "keyword",
              "index" : false
            },
            "value" : {
              "type" : "keyword",
              "index" : false
            }
          }
        },
        "parent_proprietary_id" : {
          "type" : "keyword"
        },
        "parent_title" : {
          "type" : "keyword",
          "index" : false
        },
        "parent_uri" : {
          "type" : "text",
          "index" : false
        },
        "platform" : {
          "type" : "keyword"
        },
        "print_issn" : {
          "type" : "keyword",
          "index" : false
        },
        "proprietary_id" : {
          "type" : "keyword"
        },
        "protocol" : {
          "type" : "keyword",
          "index" : false
        },
        "publication_date" : {
          "type" : "date",
          "format" : "yyyy-MM-dd"
        },
        "publication_year" : {
          "type" : "keyword"
        },
        "publisher_id" : {
          "type" : "long"
        },
        "publisher_identifiers" : {
          "properties" : {
            "type" : {
              "type" : "keyword",
              "index" : false
            },
            "value" : {
              "type" : "keyword",
              "index" : false
            }
          }
        },
        "publisher_name" : {
          "type" : "keyword",
          "index" : false
        },
        "publisher_proprietary_id" : {
          "type" : "keyword"
        },
        "referer" : {
          "type" : "keyword",
          "index" : false
        },
        "section_type" : {
          "type" : "keyword",
          "index" : false
        },
        "size" : {
          "type" : "integer",
          "index" : false
        },
        "status_code" : {
          "type" : "keyword",
          "index" : false
        },
        "tags" : {
          "type" : "keyword"
        },
        "timestamp" : {
          "type" : "date",
          "format" : "yyyy-MM-dd'T'HH:mm:ss[.SSSSSS][XXX]"
        },
        "title" : {
          "type" : "keyword",
          "index" : false
        },
        "uri" : {
          "type" : "keyword",
          "index" : false
        },
        "user_agent" : {
          "type" : "keyword"
        }
      }
    }
  }
}

[vincenbr (Vincent Bréhin)]

Hi Fabio,
The behaviour you observe is normal. You can update a static mapping to add a field (whether or not it is part of the source of existing documents), as you did, but it will not reprocess all your already indexed documents to index this field. You must explicitely reindex your documents for that.
Either with _reindex (in a new index) or with update_by_query (in place)

[Fabio_Batalha]

Thanks @vincenbr