ILM and moving shards across nodes

Elastic Stack Elasticsearch
1

Hi There,

I have a environment with multi nodes.

3 Master with with master and remote_cluster_client role
3 Data with data role
3 Warm nodes with data_warm role
3 Cold nodes with data_cold role
1 ML nodes with ml role
1 Ingest nodes with ingest role

I have configured ILM to move data in x number of days to Hot/ Warm/ Cold nodes.

when I look at the index it has applied the ILM policy however it still in the data node and haven't moved the primary or replica shards to the relevant nodes.

Any reason that it doesn't move the shards ?

Other issue is I am trying to relocate the shards manually using /_cluster/reroute, initially it shows that it's trying to relocate, however after sometime shards are in the same node.

What am I doing wrong ? do I need to do anything else ?

Any feedback appreciated.

2

For an explanation of the reasons why shards are allocated as they are in the cluster, use the cluster allocation explain API. If you need help interpreting its output, feel free to share the output (in full) here and we'll do our best to help.

3

I am getting a 400 for GET _cluster/allocation/explain

Output:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "No shard was specified in the request which means the response should explain a randomly-chosen unassigned shard, but there are no unassigned shards in this cluster. To explain the allocation of an assigned shard you must specify the target shard in the request."
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "No shard was specified in the request which means the response should explain a randomly-chosen unassigned shard, but there are no unassigned shards in this cluster. To explain the allocation of an assigned shard you must specify the target shard in the request."
  },
  "status": 400
}
4

Not sure what else to say here, the message you shared (together with the docs I linked above) say how to proceed.

5

I opened Add ref docs links to allocation explain text by DaveCTurner · Pull Request #110571 · elastic/elasticsearch · GitHub to add the missing docs link to the exception message.