How to enable the mapper-size plugin within an INDEX TEMPLATE

[ericjogonzalez (Eric Gonzalez)]

Hello, I am struggling to enable the mapper-size plugin within an index template in kibana.

A bit of background:

The mapper-size plugin is installed on all ES hosts on the cluster
All hosts have been restarted since the plugin install
I put _size in the metadata fields setting (under Kibana/Advanced Settings)
I've looked around and online all I've been able to find in regards to getting this plugin running is this page here: Using the _size field | Elasticsearch Plugins and Integrations [7.10] | Elastic

This page only describes how to enable the plugin for one, already existing index / create an index with that setting in place, this is not the method I would like to use.

My setup requires new indices created daily, therefore I need to have this change done within an index template so all new indices created daily have this done automatically, not just a one-off index.

I'm looking for anybody (ideally somebody from elastic) to give me an answer on how to best implement this change, within an index template only please!

If it helps I'm running an ELK stack on version 7.8.

[warkolm (Mark Walkom)]

Welcome to our community!

What have you tried so far, can you share a template?

[ericjogonzalez (Eric Gonzalez)]

Here is the logstash template:

  {
      "logstash" : {
        "order" : 0,
        "version" : 60001,
        "index_patterns" : [
          "logstash_*"
        ],
        "settings" : {
          "index" : {
            "number_of_shards" : "16",
            "number_of_replicas" : "1",
            "refresh_interval" : "5s"
          }
        },
        "mappings" : {
          "_meta" : { },
          "_source" : { },
          "dynamic_templates" : [
            {
              "message_field" : {
                "path_match" : "message",
                "mapping" : {
                  "norms" : false,
                  "type" : "text"
                },
                "match_mapping_type" : "string"
              }
            },
            {
              "string_fields" : {
                "mapping" : {
                  "norms" : false,
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "ignore_above" : 256,
                      "type" : "keyword"
                    }
                  }
                },
                "match_mapping_type" : "string",
                "match" : "*"
              }
            }
          ],
          "properties" : {
            "@timestamp" : {
              "type" : "date"
            },
            "geoip" : {
              "dynamic" : true,
              "type" : "object",
              "properties" : {
                 "ip" : {
                "type" : "ip"
                },
                "latitude" : {
                  "type" : "half_float"
                },
                 "location" : {
                  "type" : "geo_point"
                },
                "longitude" : {
                  "type" : "half_float"
                }
              }
            },
             "@version" : {
               "type" : "keyword"
             }
          }
        },
        "aliases" : { }
      }
    }

I used curl -XPUT "https://elastic:XXXX@es-logstash.mydomain.com:9200/logstash_dev-2021.02.04/_mappings" -H 'Content-Type: application/json' -d '{"_size": {"enabled": true}}}' to enable the size mapping on todays index. I then went into Kibana/Index Patterns and clicked on the "Refresh field list" button and _size shows up in there. I also under Kibana/Index Management went to "Manage V/Refresh index"

When I do a query like

curl -XGET "https://elastic:XXXX@es-logstash.mydomain.com:9200/logstash_dev-2021.02.04/_search" -H 'Content-Type: application/json' -d@/tmp/sq.json` with sq.json = `{ "query": { "range": { "_size": {"gt": 10 } } }, "script_fields": { "size": { "script": "doc['_size']" } } }

I get results like

    "total": {
      "value": 10000,
      "relation": "gte"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "logstash_dev-2021.02.04",
        "_type": "_doc",
        "_id": "1VjzancBGd_I3ncXIyWN",
        "_score": 1,
        "fields": {
          "size": [
            308
          ]
        }
      },

When I go to single documents such as https://kibana.mydomain.com/s/dev/app/kibana#/discover/doc/04d4bdc0-eeaf-11ea-a9a7-cb52c38ca10e/logstash_dev-2021.02.04?id=1VjzancBGd_I3ncXIyWN the _size field doesn't show up, but you can query in Kibana such as "_size > 500"

How to add this mapping to an index template is still an open question.

[ericjogonzalez (Eric Gonzalez)]

bump ?