Template for elastic indexing

Beuhlet_Reseau · April 6, 2017, 8:39am

Hello

I being created template to increase performance of indexing (and gain of space).

Where i can put not analyze string type ? (text & keyword from ES 5.x). The best will be not analyze all text field i think.

So, maybe I don't need metafield as source, score, beats info ? ... where i can delete few of them ?

 {
    "order": 0,
    "version": 50001,
    "template": "ta-test-edr",
    "settings": {
      "index": {
        "number_of_replicas": 0,
        "number_of_shards" : 1,
        "refresh_interval": "-1"
      }
    },
    "mappings": {
      "_default_": {
        "dynamic_templates": [
          {
            "string_fields": {
              "mapping": {
                "norms": false,
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword"
                  }
                }
              },
              "match_mapping_type": "string",
              "match": "*"
            }
          }
        ],
        "_all": {
          "norms": false,
          "enabled": true
        },
        "properties": {
          "@timestamp": {
            "include_in_all": false,
            "type": "date"
          },
          "geoip": {
            "dynamic": true,
            "properties": {
              "ip": {
                "type": "ip"
              },
              "latitude": {
                "type": "half_float"
              },
              "location": {
                "type": "geo_point"
              },
              "longitude": {
                "type": "half_float"
              }
            }
          },
          "@version": {
            "include_in_all": false,
            "type": "keyword"
          }
        }
      }
    },
    "aliases": {}
  }

Beuhlet_Reseau · April 6, 2017, 12:43pm

I believe it's like to delete field not use :

"properties": {
            "source" : { "enabled" : false },
            "beat": { "enabled" : false },
            "@version": { "enabled" : false },
            "name": { "enabled" : false },
            "version": { "enabled" : false },
            "host": { "enabled" : false },
            "input_type": { "enabled" : false },
            "tags": { "enabled" : false },
            "type": { "enabled" : false } }

Beuhlet_Reseau · April 6, 2017, 2:56pm

Ok i belive it's work fine with this template :

PUT _template/template_edr
{
    "order": 0,
    "version": 50001,
    "template": "edr-*",
    "settings": {
      "index": {
        "number_of_replicas": 0,
        "number_of_shards" : 1,
        "refresh_interval": "-1"
      }
    },
    "mappings": {
      "_default_": {
        "dynamic_templates": [
          {
            "string_fields": {
              "mapping": {
                "norms": false,
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "index": "not_analyzed"
                  }
                }
              },
              "match_mapping_type": "string",
              "match": "*"
            }
          }
        ],
        "_all": {
          "norms": false,
          "enabled": true
        },
        "properties": {
          "edr_Granctets": {
            "type": "long"
          },
          "edr_EI": {
            "type": "long"
            },
          "edr_Janomer": {
            "type": "byte"
          },
          "edr_MC": {
            "type": "integer"
            },
          "edr_DN": {
            "type": "long"
            },
          "edr_MasDN": {
            "type": "long"
            },
          
          "edr_ModonFlag": {
            "type": "byte"
            },
          "edr_ParlFlag": {
            "type": "byte"
            },
          "edr_SurId": {
            "type": "long"
          },
          "edr_Usimit": {
            "type": "long"
          },
          "edr_Usetets": {
            "type": "long"
          },
          
          "@timestamp": {
            "include_in_all": false,
            "type": "date"
          },
          "geoip": {
            "enabled": false
            },
            "source" : { 
              "enabled" : false 
            },
            "beat": { 
              "enabled" : false 
            },
            "@version": {
              "include_in_all": false,
              "type": "keyword"
             },
            "name": { 
              "enabled" : false 
            },
            "host": { 
              "enabled" : false 
            },
            "input_type": { 
              "enabled" : false 
            },
            "tags": { 
              "enabled" : false 
            },
            "type": { 
              "enabled" : false 
              
            }
        }
      }
    },
    "aliases": {}
  }

How to know if my settings are correctly understand ? thank you

Beuhlet_Reseau · April 7, 2017, 3:40pm

hummm i have some errors in elastic log :

[2017-04-07T17:28:45,155][WARN ][o.e.d.i.m.StringFieldMapper$TypeParser] The [string] field is deprecated, please use [text] or [keyword] instead on [uiStateJSON]
[2017-04-07T17:28:45,155][WARN ][o.e.d.i.m.StringFieldMapper$TypeParser] The [string] field is deprecated, please use [text] or [keyword] instead on [description]
[2017-04-07T17:28:45,156][WARN ][o.e.d.i.m.StringFieldMapper$TypeParser] The [string] field is deprecated, please use [text] or [keyword] instead on [savedSearchId]
[2017-04-07T17:28:45,156][WARN ][o.e.d.i.m.StringFieldMapper$TypeParser] The [string] field is deprecated, please use [text] or [keyword] instead on [searchSourceJSON]
[2017-04-07T17:28:45,156][WARN ][o.e.d.i.m.StringFieldMapper$TypeParser] The [string] field is deprecated, please use [text] or [keyword] instead on [title]
[2017-04-07T17:28:45,156][WARN ][o.e.d.i.m.StringFieldMapper$TypeParser] The [string] field is deprecated, please use [text] or [keyword] instead on [visState]
[2017-04-07T17:28:45,157][WARN ][o.e.d.i.m.StringFieldMapper$TypeParser] The [string] field is deprecated, please use [text] or [keyword] instead on [uiStateJSON]
[2017-04-07T17:28:45,157][WARN ][o.e.d.i.m.StringFieldMapper$TypeParser] The [string] field is deprecated, please use [text] or [keyword] instead on [description]
[2017-04-07T17:28:45,157][WARN ][o.e.d.i.m.StringFieldMapper$TypeParser] The [string] field is deprecated, please use [text] or [keyword] instead on [savedSearchId]
[2017-04-07T17:28:45,157][WARN ][o.e.d.i.m.StringFieldMapper$TypeParser] The [string] field is deprecated, please use [text] or [keyword] instead on [searchSourceJSON]

So I use text in my template no ?

warkolm · April 8, 2017, 11:07pm

That is what you have here;

 "string_fields": {
              "mapping": {
                "norms": false,
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword"
                  }
                }
          }

Don't send them is the easiest way But you can disable source (as it is an ES native field) if you really want.

Beuhlet_Reseau · April 10, 2017, 8:35am

@warkolm

"source" : {
"enabled" : false
},

That is ? or you talk to remove source field directly in logstash with the remove_field ?
So, i want remove field like input_type, beat.*, _score,_id ??? (I have disable it in template but i continu to see them in discover)

About depracated error, why ? I use text and keyword no ? @Christian_Dahlqvist

warkolm · April 10, 2017, 10:01am

That.

You cannot remove _id, if that is what you mean.

Please don't ping people like that.

Have a look at https://www.elastic.co/guide/en/elasticsearch/reference/5.3/breaking_50_mapping_changes.html#_literal_string_literal_fields_replaced_by_literal_text_literal_literal_keyword_literal_fields

Beuhlet_Reseau · June 13, 2019, 8:53am

I understand @warkolm

If i do that :

mutate {
                remove_field => [ "message", "beat", "input_type", "type", "tags", "host" ]
                   }

It's also good no ?

So, I have this error from few moment :

it's because of template (see above) when i delete my template i haven't error

billnbell · April 12, 2017, 3:08pm

Shouldn't the default be norms: false on all keyword fields?

Beuhlet_Reseau · April 13, 2017, 2:42pm

Yes i copy it from default logstash template (GET /template).

Why ??

So I see every fields (numeric and text) in index pattern page but i see always this error message on discover

Beuhlet_Reseau · April 14, 2017, 9:16am

IF i want not analyze few numeric data fields i have just put :

"numeric_field" {
type="long"
index="not_analyze"
}

?

system · May 12, 2017, 9:20am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.