The question is still regarding templates and why when setting settings through a template they get applied to the index but not necessarily read by ES.
12 Data Nodes 64 GB RAM, 6TB HDD 24 core (box_type: hot)
3 Data Nodes 64 GB RAM, A LOT of storage (spinning) (box_type: cold)
5 Client Nodes 32 GB RAM 24 core
What my scenario is:
I have data indexed daily, I want to age the data at 30 day from node box_type: hot to node box_type: cold. I have a template set up to match all indices '*' that creates the setting:
I want this to set my shards/replicas, along with my routing allocation.
What do I see happening:
When the index gets created first the primary shards do not allocate, I receive a red cluster at midnight UTC time when net indices are created because all primary shards become unalloocated. When I look at the index setting of the unallocated shards I can see the setting are set in one index but not the other 3 but elastic seems to ignore the setting on the one index.
When I run curator to set the routing allocation I can see a second setting for shards, replicas, and routing allocation on the first index, and I can see the settings get applied a first time on the other indices. After a few seconds I get a green cluster because the settings are applied and routing happens according to the settings.
I don't know why the template is seeming ignored for 3 of my 4 indices. I don't know why it is applied on one index but not used by elastic. My goal would be to have my indices created with box_type set to hot and replicas and shards set appropriately and my cluster stay green the entire time so I am not left with unallocated shards.
As stated I have a hacky work around. I run a sloppy script to run curator every day at midnight 05, all shards get reallocated and I am good. But I wouldn't think this is necessary seeing that I created a template with the settings therein.