Dynamic Template not working

Elastic Stack Elasticsearch
1

I want to apply custom analyzer to all fields in an existing mapping to enable search with special characters.
I created custom ngram analyzer and a dynamic template to apply custom_analyzer on all text fields to do so.
Search for simple text without any special character works fine But when I search for any special character within _category field , it is returning 0 results.

To test for custom_analyzer I created a new field(email) in the mapping with analyzer and search of special characters on this email field giving me expected results.
I also created a new field(phone No) without analyzer to check if dynamic_template apply my custom_analyzer to that phone field , But I am not getting expected results.

Please suggest how dynamic_templates work? If we can apply dynamic_templates to existing mappings or if to a new field in existing mapping?

http://localhost:9200/audit/_mapping

"audit": {
"mappings": {
"logs": {
"dynamic_templates": [
{
"ngram_string_analyzers": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"analyzer": "custom_analyzer",
"type": "text"
}
}
}
],
"properties": {
"_category": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"_createdOn": {
"type": "date"
}
}
}
}

}

"audit": {
"settings": {
"index": {
"number_of_shards": "5",
"provided_name": "audit",
"creation_date": "1556778930604",
"analysis": {
"analyzer": {
"custom_analyzer": {
"type": "custom",
"tokenizer": "my_tokenizer"
}
},
"tokenizer": {
"my_tokenizer": {
"type": "ngram",
"min_gram": "1",
"max_gram": "10"
}
}
},
"number_of_replicas": "1",
"version": {
"created": "6040099"
}
}
}
}
}

2

Hi,

I think the document is explicit and can help you to get the concept.

About phone number mapping here a part of what I use but there's some specific things related to Japanese language, I tried to clean but just to get the concept, I copied it from somewhere but can't found back the source.

{
    "index_patterns" : "appli1_*",
    "order" : 1,
    "settings" : {
      "analysis" : {
            "filter" : {
"phone_number": {
                  "type": "pattern_capture",
                  "preserve_original": true,
                  "patterns": [
                    "\\d*"
                  ]
                },
                "smallest_digit": {
                  "type": "length",
                  "min": 6
                },
                "not_empty": {
                  "type": "length",
                  "min": 1
                }
            },
  "digits_only": {
                "type": "pattern_replace",
                "pattern": "[^\\d]"
              }
            },
"analyzer" : {
"my_phone_number_analyzer" : {
                  "char_filter": ["icu_normalizer", "digits_only"],
                    "tokenizer": "YOUR TOKENIZER",
                    "filter": ["phone_number", "smallest_digit"]
                },
                "my_phone_number_search_analyzer" : {
                  "char_filter": ["icu_normalizer", "digits_only"],
                  "tokenizer" : "YOUR TOKENIZER",
                  "filter" : ["not_empty"]
                }

In my case I call all my fields that have phone number with a _number suffix

"mappings" : {
        "doc" : {
            "dynamic_templates" : [
{"phone_number": {
                  "match" : "*_number",
                  "mapping" : {
                    "type" :  "text",
                    "fields":{
                      "untouched":{
                        "type": "keyword",
                        "ignore_above": 256
                      },
                      "my_phone_number_analyzer":{
                        "type": "text",
                        "analyzer": "my_phone_number_analyzer",
                        "search_analyzer" : "my_phone_number_search_analyzer"
                      },
                      "my_ngram_analyzer":{
                        "type": "text",
                        "analyzer": "my_ngram_analyzer"
                      }
                    }
                  }
                }}

Note that if you use elastic >= 7 you need to remove the "doc" key as the type no more exist. (Not fully sure as I didn't migrate yet but you may get error if you copy paste).

Found back the source it's about US number as I said before I adapt it for Japanese so depend on your case maybe you 'll better to start from the original blog post:

Hope it help.

3

Hi Gabriel,

Thanks for your quick reply.
But phone No field is just an example, I have around 20 text fields that can have special characters in values. So I need to apply one custom analyzer to all fields. Also My mapping is dynamic and any text field can be added later also. So I want to make sure that each new field will also contain my custom analyzer.
The only way I know is using dynamic templates and it is not working in my case. So, please help me if there is any other way to implement a custom analyzer on all text fields or if you can help me in dynamic templates.

4

Sorry I didn't get your problem, let me try again.

You create a field email something like "aaa@aa.com" which is working because it's a string and in your template you defined

"ngram_string_analyzers": {
"match": "*",
"match_mapping_type": "string",

the mapping apply, no problem BUT

this field is certainly 00112313213 <-- which is not a string, but you didn't tell which mapping apply so I try to guess and it's certainly something like long.

So if you try to remove the "match_mapping_type": "string", it will apply to all your fields if it's what you want?

5

Hi Gabriel,

My issue got resolved after changing the mapping of dynamic template to -
"dynamic_templates": [
{
"ngram_string_analyzers": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"analyzer": "custom_analyzer",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
}
]

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.