[closed] Template with not_analyzed fields by default: how to add .raw field to specific analyzed fields

Elastic Stack Elasticsearch
1

Hello,

I'm using ELK to centralize events.
I customized my Elasticsearch template to have not_analyzed fields by default, and some specific analyzed fields, but now I see that I am blocked to create some vizualisations because these fields have no .raw field.

I want to keep my actual template but also have .raw not_analyzed fields for all my analyzed ones. How can I do that?

My template looks like this

{
  "template" : "events-*",
  "mappings" : {
    "_default_" : {
       "_all" : {"enabled" : true},
       "dynamic_templates" : [ {
         "not_analyzed_string_fields" : {
           "match" : "*",
           "match_mapping_type" : "string",
           "mapping" : {
             "type" : "string", "index" : "not_analyzed", "omit_norms" : true, "doc_values" : true
           }
         }
       } ],
       "properties" : {
         "@version": { "type": "string", "index": "not_analyzed" },
         "action" : {
             "type" : "string",
             "index" : "analyzed"
         },
         "message" : {
             "type" : "string",
             "index" : "analyzed"
         },
         "path" : {
             "type" : "string",
             "index" : "analyzed"
         }
       }
    }
  }
}

I imagine there can be 2 answers:

  • add the .raw field for each of the specifically analyzed fields
  • tell ES to automatically add this .raw field for each analyzed field

I would prefer solution 2 but I have no idea how to do it. Neither for solution 1.

Thanks for your help

2

Logstash does this sort of thing (automatically creating .raw not_analyzed fields for each string (analyzed) field. You can see how it does it over here: https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch/elasticsearch-template.json#L19-L29

3

Won't this code create a .raw field for each field, be it analyzed or not?
I'll try and see.

4

Apologies; I think I misunderstood your question. You want to create .raw not_analyzed fields only for analyzed string fields. So, if a field is already not_analyzed, you don't want to create a .raw field for it.

You could obviously do this with option 1. Alternatively, if you really want this to be dynamic, you can name your analyzed string fields a certain way, for example with the prefix "analyzed_" and then use "match": "analyzed_*" in your dynamic template definition.

5

Okay I understand the problem with the match:'*' and how using a special name could solve it.
I won't change their name, so I'll go for solution 1, explicitely telling that I want a .raw not_analyzed field for each analyzed field.
Do you have some example code to do this?

edit
I found how, my field definitions looks like this now

"action" : {
         "type" : "string",
         "index" : "analyzed",
         "fields" : {
               "raw" : {"type": "string", "index" : "not_analyzed", "doc_values" : true}
         }
},