# ILM hot/warm policy is not moving shards

**URL:** <https://discuss.elastic.co/t/ilm-hot-warm-policy-is-not-moving-shards/272603>\
**Category:** Elasticsearch\
**Tags:** ilm-index-lifecycle-management\
**Created:** [May 10, 2021, 5:15pm UTC](https://discuss.elastic.co/t/ilm-hot-warm-policy-is-not-moving-shards/272603 "2021-05-10T17:15:57Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![vkatsikaros](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vkatsikaros/32/110600_2.png) [@vkatsikaros](https://discuss.elastic.co/u/vkatsikaros)\
**Post date:** [May 10, 2021, 5:15pm UTC](https://discuss.elastic.co/t/ilm-hot-warm-policy-is-not-moving-shards/272603/1 "2021-05-10T17:15:57Z")

</div>

Hi

I have created an ILM policy and template, and I create daily indices with Logstash on an ES cluster 7.10. I have some fast and expensive nodes as `hot` (and other types) and a few slow and inexpensive as `warm` only. I expected that my ILM policy would move the in the `warm` phase to the `warm` nodes. But this is not happening (apparently I do something wrong).

# Nodes

Some nodes are `dhilrst` (ie `h` `hot` included) and a some are `w` (`warm`) only.

```auto
$ curl localhost:9200/_cat/nodes?v
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
...
10.40.21.141 34 96 6 0.67 0.56 0.66 dhilrst - ip-10-40-21-141
10.40.21.136 43 92 1 0.07 0.02 0.03 dhilrst - ip-10-40-21-136
10.40.23.24 43 94 6 0.18 0.30 0.41 dhilrst - ip-10-40-23-24
10.40.21.234 30 99 3 0.00 0.08 0.09 w - ip-10-40-21-234
10.40.23.37 56 99 4 0.02 0.05 0.08 w - ip-10-40-23-37
10.40.22.135 50 99 4 0.00 0.02 0.07 w - ip-10-40-22-135
...

```

# ILM is ebabled

```auto
$ curl -s 'localhost:9200/_ilm/status' | jq .
{
  "operation_mode": "RUNNING"
}

```

# No Cluster-level shard allocation and routing settings

```auto
$ curl -s localhost:9200/_cluster/settings | jq .
{
  "persistent": {
    "xpack": {
      "monitoring": {
        "elasticsearch": {
          "collection": {
            "enabled": "false"
          }
        }
      }
    }
  },
  "transient": {
    "cluster": {
      "routing": {
        "allocation": {
          "include": {
            "_ip": ""
          },
          "exclude": {
            "_ip": ""
          }
        }
      }
    }
  }
}

```

# ILM policy and template

I expect that this policy does the following:

- index age: 0ms - 1day =\> phase hot (index on `hot` node)
- index age: 1 day - 21 days =\> phase warm (index on `warm` node)
- index age: 21 days - inf =\> phase delete (index gets deleted)

```auto
$ curl -s localhost:9200/_ilm/policy/foo?pretty | jq .foo
{
  "version": 5,
  "modified_date": "2021-04-13T15:29:32.797Z",
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {}
      },
      "delete": {
        "min_age": "21d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      },
      "warm": {
        "min_age": "1d",
        "actions": {
          "migrate": {
            "enabled": true
          }
        }
      }
    }
  }
}

```

Index template

```auto
$ curl -s 'localhost:9200/_index_template' | jq '.index_templates[] | select(.name=="foo")'
{
  "name": "foo",
  "index_template": {
    "index_patterns": [
      "foo.logstash-*"
    ],
    "template": {
      "settings": {
        "index": {
          "lifecycle": {
            "name": "foo"
          }
        }
      }
    },
    "composed_of": []
  }
}

```

# "today" (2021.05.10) index

Today's index is `data_content` (by default) so it does get assigned to the `content` node.

```auto
$ curl -s 'localhost:9200/foo.logstash-2021.05.10' | jq '."foo.logstash-2021.05.10".settings'
{
  "index": {
    "lifecycle": {
      "name": "foo"
    },
    "routing": {
      "allocation": {
        "include": {
          "_tier_preference": "data_content"
        }
      }
    },
    "number_of_shards": "1",
    "provided_name": "foo.logstash-2021.05.10",
    "creation_date": "1620604801015",
    "number_of_replicas": "1",
    "uuid": "J9uINSGSSuWjO60xgO_PRA",
    "version": {
      "created": "7100199"
    }
  }
}

```

Indeed, the index's shards are on `dhilrst` nodes.

```auto
ubuntu@logs-live-master-us-west-2a:~$ curl -s 'localhost:9200/_cat/shards' | grep foo.logstash-2021.05.10
foo.logstash-2021.05.10 0 p STARTED 30069059 38gb 10.40.21.141 ip-10-40-21-141
foo.logstash-2021.05.10 0 r STARTED 30037707 36.4gb 10.40.23.24 ip-10-40-23-24

```

ILM-wise the index is on its `hot` phase, without any errors

```auto
$ curl -s 'localhost:9200/foo.logstash-2021.05.10/_ilm/explain?pretty'
{
  "indices" : {
    "foo.logstash-2021.05.10" : {
      "index" : "foo.logstash-2021.05.10",
      "managed" : true,
      "policy" : "foo",
      "lifecycle_date_millis" : 1620604801015,
      "age" : "16.75h",
      "phase" : "hot",
      "phase_time_millis" : 1620604828556,
      "action" : "complete",
      "action_time_millis" : 1620604826702,
      "step" : "complete",
      "step_time_millis" : 1620604828556,
      "phase_execution" : {
        "policy" : "foo",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : { }
        },
        "version" : 5,
        "modified_date_in_millis" : 1618327772797
      }
    }
  }
}

```

# yesrerday's (2021.05.09) index (where the problem is shown)

Yesterday's index is `data_warm,data_hot`.

```auto
$ curl -s 'localhost:9200/foo.logstash-2021.05.09' | jq '."foo.logstash-2021.05.09".settings'
{
  "index": {
    "lifecycle": {
      "name": "foo"
    },
    "routing": {
      "allocation": {
        "include": {
          "_tier_preference": "data_warm,data_hot"
        }
      }
    },
    "number_of_shards": "1",
    "provided_name": "foo.logstash-2021.05.09",
    "creation_date": "1620518400615",
    "number_of_replicas": "1",
    "uuid": "SSHXGJZmS52jC1gpakOIyw",
    "version": {
      "created": "7100199"
    }
  }
}

```

I'd expect the index to be on a `warm` node, BUT it is not.... it's on `dhilrst` nodes. **This is where the problem is shown.**

```auto
$ curl -s 'localhost:9200/_cat/shards' | grep foo.logstash-2021.05.09
foo.logstash-2021.05.09 0 p STARTED 40757640 48.2gb 10.40.23.24 ip-10-40-23-24
foo.logstash-2021.05.09 0 r STARTED 40757640 48.2gb 10.40.21.136 ip-10-40-21-136

```

ILM-wise the index is in its `warm` phase without any errors

```auto
$ curl -s 'localhost:9200/foo.logstash-2021.05.09/_ilm/explain?pretty'
{
  "indices" : {
    "foo.logstash-2021.05.09" : {
      "index" : "foo.logstash-2021.05.09",
      "managed" : true,
      "policy" : "foo",
      "lifecycle_date_millis" : 1620518400615,
      "age" : "1.69d",
      "phase" : "warm",
      "phase_time_millis" : 1620604818008,
      "action" : "complete",
      "action_time_millis" : 1620604829692,
      "step" : "complete",
      "step_time_millis" : 1620604829692,
      "phase_execution" : {
        "policy" : "foo",
        "phase_definition" : {
          "min_age" : "1d",
          "actions" : {
            "migrate" : {
              "enabled" : true
            }
          }
        },
        "version" : 5,
        "modified_date_in_millis" : 1618327772797
      }
    }
  }
}

```

# Replicate settings on a docker test cluster

I have replicated the settings on a docker ES 7.10 cluster and the shards move to the `warm` node as expected.

---

<div class="post-metadata">

**Author:** ![andreidan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andreidan/32/78167_2.png) [@andreidan](https://discuss.elastic.co/u/andreidan)\
**Post date:** [May 11, 2021, 9:30am UTC](https://discuss.elastic.co/t/ilm-hot-warm-policy-is-not-moving-shards/272603/2 "2021-05-11T09:30:53Z")

</div>

Thanks for using Elasticsearch.

The shards are not moving to the warm tier becuase your hot tier **also** contains the generic [_data_ role](https://www.elastic.co/guide/en/elasticsearch/reference/7.12/modules-node.html#data-node). The data role can host any specialised data tier roles (so your `dhilrst` nodes can host the warm tier too due to the `d` role).

You'd have to remove the generic `data` role from your node roles configurations for the hot tier to only host hot and content data.

---

<div class="post-metadata">

**Author:** ![vkatsikaros](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vkatsikaros/32/110600_2.png) [@vkatsikaros](https://discuss.elastic.co/u/vkatsikaros)\
**Post date:** [May 11, 2021, 9:56am UTC](https://discuss.elastic.co/t/ilm-hot-warm-policy-is-not-moving-shards/272603/3 "2021-05-11T09:56:26Z")

</div>

@andreidan thanks for the prompt reply.

I understand you suggest that we remove the generic `data` role from **all** the nodes of the cluster. I would like to ask how will this affect indices that are not managed with the data tiers (`warm`/`hot`), both new and existing indices.

Let me give some examples:

# index created AFTER we enabled ILM policies:

Indices are created automatically from Logstash. In this example the index has the default (?) `"_tier_preference": "data_content"`

```auto
$ curl -s 'localhost:9200/lala-2021.w06/_ilm/explain?pretty'
{
  "indices" : {
    "lala-2021.w06" : {
      "index" : "lala-2021.w06",
      "managed" : false
    }
  }
}

$ curl -s 'localhost:9200/lala-2021.w06/' | jq '."lala-2021.w06".settings'
{
  "index": {
    "routing": {
      "allocation": {
        "include": {
          "_tier_preference": "data_content"
        }
      }
    },
    "number_of_shards": "1",
    "provided_name": "lala-2021.w06",
    "creation_date": "1612742422237",
    "number_of_replicas": "1",
    "uuid": "R804zhniR7SMo7_QAePH0g",
    "version": {
      "created": "7100199"
    }
  }
}

```

# index created BEFORE we enabled ILM policies:

We have indices that were created before the introduction of ILM. In this example, the index has no routing allocation at all

```auto
$ curl -s 'localhost:9200/lala-2020.w40/_ilm/explain?pretty'
{
  "indices" : {
    "lala-2020.w40" : {
      "index" : "lala-2020.w40",
      "managed" : false
    }
  }
}

$ curl -s 'localhost:9200/lala-2020.w40/' | jq '."lala-2020.w40".settings'
{
  "index": {
    "creation_date": "1601856015556",
    "number_of_shards": "1",
    "number_of_replicas": "1",
    "uuid": "UsUCSLCFSomidJ2WnSa83Q",
    "version": {
      "created": "7070199",
      "upgraded": "7100199"
    },
    "provided_name": "lala-2020.w40"
  }
}

```

---

<div class="post-metadata">

**Author:** ![andreidan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andreidan/32/78167_2.png) [@andreidan](https://discuss.elastic.co/u/andreidan)\
**Post date:** [May 11, 2021, 10:25am UTC](https://discuss.elastic.co/t/ilm-hot-warm-policy-is-not-moving-shards/272603/4 "2021-05-11T10:25:20Z")

</div>

> [@vkatsikaros](#):
>
> In this example the index has the default (?) `"_tier_preference": "data_content"`

This is due to the automatic new index allocation [Data tiers | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/data-tiers.html#data-tier-allocation) (irrespective of being managed by ILM)

> the index has no routing allocation at all

Indices without any allocation rules will be able to reside on **any** data holding-capable node (ie. all data tier specific node roles can hold data).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 8, 2021, 10:26am UTC](https://discuss.elastic.co/t/ilm-hot-warm-policy-is-not-moving-shards/272603/5 "2021-06-08T10:26:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.