Unable to setup a normalizer on a keyword using dynamic mapping

Elastic Stack Elasticsearch
1

Hi,

I would like to add a custom normalizer on my keyword fields.
I followed up the indications in Adding a normalizer for case-insensitive search on keyword field to an active index

It works fine when I apply it to a field in the mapping definition. But as I would like this to be the defaults for strings in my index, I did the setup using a dynamic mapping.
The dynamic mapping is accepted, but when a new field is effectively added, the normalizer is not applied.

I tested it on ES 5.6.4 and 6.3.1 running on the jvm v1.8.0_60

here is a small test that reproduce the issue:

# creates a dynamic template with the normalizer + one custom field with the normalizer + a dynamic rule for new fields
PUT _template/test-analyser
{  "order": 0,
  "template": "analyser-*",
  "settings": {
    "index": {
      "analysis": {
        "normalizer": {
          "lowerascii": {
            "type": "custom",
            "filter": [
              "lowercase",
              "asciifolding"
            ]
          }
        }
      }
    }
  },
  "mappings": {
    "log": {
      "dynamic_templates": [
        {
          "standard_string_fields": {
            "match_mapping_type": "string",
            "mapping": {
              "type": "keyword",
              "normaliser": "lowerascii"
            }
          }
        }
      ],
      "properties": {
        "field_defined_in_mapping": {
          "type": "keyword",
          "normalizer": "lowerascii"
        }
      }
    }
  }
  
}

# add a doc with the existing field
PUT analyser-2018.08.22/log/1
{
  "field_defined_in_mapping":"TEST"
}

# add a doc with a new field
PUT analyser-2018.08.22/log/2
{
  "new_field": "TEST"
}

# case insensitive search on the first field
GET analyser-2018.08.22/_search?q=field_defined_in_mapping:test
# returns doc 1 (which is expected)

# case insensitive search on the new field
GET analyser-2018.08.22/_search?q=new_field:test
# returns nothing

GET analyser-2018.08.22

# in the mapping definition the normalizer is not applied to the new field
#      "log": {
#        "properties": {
#          "field_defined_in_mapping": {
#            "type": "keyword",
#            "normalizer": "lowerascii"
#          },
#          "new_field": {
#            "type": "keyword"
#          }
#        }

Is it possible to define a normalizer inside a dynamic mapping?

2

Your dynamic template is defining normaliser, with an s. Try with normalizer.

I have not tested this yet, but I would expect an exception instead of failing silently.

3

Thanks a lot, that was it, now the fields are indexed with the normaliZer applied!

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.