Elastic Template to manage Settings

Elastic Stack Elasticsearch
1

I am trying to set shards, replicas, and shard allocation utilizing a template. I thought I had everything working but newly created indices are not getting the index template applied.

my template is:

{
  "order": 2,
  "template": "*",
  "settings": {
    "index": {
      "index": {
        "routing": {
          "allocation": {
            "require": {
              "box_type": "hot"
            }
          }
        }
      },
      "refresh_interval": "5s",
      "number_of_shards": "10",
      "number_of_replicas": "1"
    }
  },
  "mappings": {},
  "aliases": {}
}

My thought was this would match all templates that are in elastic. So if any new index were created this template would be applied. I checked all existing templates and there is one Order 0 and one Order 2, so I thought this being Order 3 would be the last applied and override any other of these settings in the previous templates.

My issue is it is not working. Can someone please advise where my mistake is being made?

2

Hi,

your assumption that templates get applied with increasing order. What exactly isn't working as expected? What do the other templates look like and which setting doesn't turn out as expected when you create a new index?

3

sorry. My assumption that the templates in a higher order get applied last doesnt' seem to apply.

The setting of routing.allocation.require doesn't get set nor do shards or replicas. I dont' have any templates with a higher order. I have the basic logstash template for logstash-* indices but non e of my indices follow that pattern. When new indices get created they are going to nodes with box_type set to cold and they don't have the correct number of shards or replicas set.

I'm happy to share most any data I can to help. Just let me know whats needed

4

Hi,

to debug this you could add something unique to each of your templates (e.g. a bogus mapping with a unique name for each template you have). Then you create a new index and check which of these mappings are there. This way you know which templates are appied for this index name pattern. Remove those mappings again and add a setting with a different value for each effective template. Rinse and repeat, to see which template gets applied last.
If this doesn't work, could you try to set up a minimal example of where the settings overwrite doesn't work as expected (also add the Elasticsearch version you are using)?

5

Thank you. I will try that. I am on ES 2.3.3.

6

So still no luck. I can see the setting in the index setting section. I see the correct setting for:

 "settings": {
    "index": {
      "index": {
        "routing": {
          "allocation": {
            "require": {
              "box_type": "hot"
            }
          }
        }
      },
      "refresh_interval": "5s",
      "number_of_shards": "10",
      "number_of_replicas": "1"
    }

But it seems elastic is ignoring the setting. Why would this happen. If the setting is visible when I view "GET /index-name/_settings", then why would elastic not use it. When I run curator and tell it :
curator --loglevel DEBUG --timeout 3600 --host 10.1.55.2 allocation --rule box_type=hot indices --prefix index-name

it applies a second instance of the setting and then data gets allocated appropriately.

I created a bash script to run every day to reallocate the shards but I wouldn't think that should be necessary if I am using index templates to create the right settings.

Thanks

7

Hi,
so do I understand this correctly, this is no longer an issue with templates and the overwrite order but more a question about allocation? Can you specify again what your scenario looks like, i.e. what your expected behaviour for this setting is and what you see instead. Also how many nodes do you have?

8

The question is still regarding templates and why when setting settings through a template they get applied to the index but not necessarily read by ES.

My setup:
ES 2.3.3
12 Data Nodes 64 GB RAM, 6TB HDD 24 core (box_type: hot)
3 Data Nodes 64 GB RAM, A LOT of storage (spinning) (box_type: cold)
5 Client Nodes 32 GB RAM 24 core

What my scenario is:
I have data indexed daily, I want to age the data at 30 day from node box_type: hot to node box_type: cold. I have a template set up to match all indices '*' that creates the setting:

 "settings": {
    "index": {
      "index": {
        "routing": {
          "allocation": {
            "require": {
              "box_type": "hot"
            }
          }
        }
      },
      "refresh_interval": "5s",
      "number_of_shards": "10",
      "number_of_replicas": "1"
    }

I want this to set my shards/replicas, along with my routing allocation.

What do I see happening:
When the index gets created first the primary shards do not allocate, I receive a red cluster at midnight UTC time when net indices are created because all primary shards become unalloocated. When I look at the index setting of the unallocated shards I can see the setting are set in one index but not the other 3 but elastic seems to ignore the setting on the one index.

When I run curator to set the routing allocation I can see a second setting for shards, replicas, and routing allocation on the first index, and I can see the settings get applied a first time on the other indices. After a few seconds I get a green cluster because the settings are applied and routing happens according to the settings.

I don't know why the template is seeming ignored for 3 of my 4 indices. I don't know why it is applied on one index but not used by elastic. My goal would be to have my indices created with box_type set to hot and replicas and shards set appropriately and my cluster stay green the entire time so I am not left with unallocated shards.

As stated I have a hacky work around. I run a sloppy script to run curator every day at midnight 05, all shards get reallocated and I am good. But I wouldn't think this is necessary seeing that I created a template with the settings therein.

9

I think you have to remove first level 'index' within 'settings'. See example https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html

10

I don't quite understand what you mean by "first level index"

11

This made me suspicious as well, I think your template that you posted in the first posting has an "index" level too much as @johtani already suggested:

{
  "order": 2,
  "template": "*",
  "settings": {
    "index": { <-- This can go?? including the closing bracket
      "index": {
        "routing": {
          "allocation": {
            "require": {
              "box_type": "hot"
            }
          }
        }
      },
      "refresh_interval": "5s",
      "number_of_shards": "10",
      "number_of_replicas": "1"
    }
  },
  "mappings": {},
  "aliases": {}
}
12

ahhhh. Thank you. I will try and when things get recreated tonight I will report back

13

SOLVED

Just wanted to reply back and say thank you. Apparently my inexperience as an elastic admin showed and I made a simple syntax error that has caused me trouble for weeks.

After removing the second "index" and its brackets my index template works as expected.

Thank you for you help,