How to change the number of shards per index in a cluster where indexes are created per day

Elastic Stack Elasticsearch
#1

We have an ES cluster storing events for about 10 different applications. Each applications events are stored in an index specific to that application (the apps are unrelated), and a new index is created daily for each application (i.e. app1-2016.05.04, app1-2016.05.05, app2-2016.05.04........etc.). Creation of the indices for some of the applications use templates, some just take the ES defaults. In all cases we're using the default of 5 primary shards, however we want to lower that to 3 primary shards per index/per day.

How do we make this update so that the "next" days indices, and all future indices get created with only the 3 primary shards? I know you can't make this static update to the already-created indexes without going through a re-indexing, so how do we change both the default index template and our custom templates so that at 12:00am, when the next days indexes are created they only have 3 shards?

Appreciate your feedback.

#2

templates?

#3

Hi Nik - here's one of the app templates. I think i figured it out though. You have to download a copy of the template, edit it, then re-XPUT the template using the file as json input. Am i correct?

{
  "suprv_template" : {
    "order" : 0,
    "template" : "suprv-*",
    "settings" : {
      "index" : {
        "refresh_interval" : "5s"
      }
    },
    "mappings" : {
      "_default_" : {
        "dynamic_templates" : [ {
          "message_field" : {
            "mapping" : {
              "index" : "analyzed",
              "omit_norms" : true,
              "type" : "string"
            },
            "match_mapping_type" : "string",
            "match" : "message"
          }
        }, {
          "string_fields" : {
            "mapping" : {
              "index" : "not_analyzed",
              "omit_norms" : true,
              "type" : "string"
            },
            "match_mapping_type" : "string",
            "match" : "*"
          }
        } ],
        "_all" : {
          "omit_norms" : true,
          "enabled" : true
        },
        "properties" : {
          "geoip" : {
            "dynamic" : true,
            "type" : "object",
            "properties" : {
              "location" : {
                "type" : "geo_point"
              }
            }
          },
          "@version" : {
            "index" : "not_analyzed",
            "type" : "string"
          }
        }
      }
    },
    "aliases" : { }
  }
}
#4

Yup. You'd GET the template, edit it, and PUT it back.

#5

TY Nik. Appreciate.

#6

Well, this worked for those indexes that are created by specific templates. However many of our indexes are just created with ES defaults. How do we go about changing the actual ES default for # of primary shards?

#7

I think the normal thing is to have a template that matches all indexes.
I'm honestly not totally sure though. Sorry!

#8

You specify the default number of shards and replicas in your elasticsearch.yml file.

index.number_of_shards: 3
index.number_of_replicas: 0

If there is no matching templates for an index, the default applies. To change number of shards and replica for existing indices, you will need to reindex.

#9

TY. This did resolve for those indices not created from template.

#10