ECK cluster always on MigratingData status

[chenyg0911 (Chan)]

eck: v1.8.
kubernetes: v1.20.7

kubectl get es:
NAME HEALTH NODES VERSION PHASE AGE
es-v7132 yellow 4 7.13.2 MigratingData 83d

GET _cat/health?v
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1634523756 02:22:36 es-v7132 yellow 4 3 43 23 0 0 3 0 - 93.5%

GET _cluster/settings

{
  "persistent" : { },
  "transient" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "exclude" : {
            "_name" : "es-v7132-es-datas-2"
          }
        }
      }
    }
  }
}

Have tried:

PUT _cluster/settings

{
  "transient" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "exclude" : {
            "_name" : null
          }
        }
      }
    }
  }
}

All shards will sync correctlly.
ButAfter a few minutes, The cluster healthy will turn back to yellow, some shards can't replicate, the eck cluster status MigratingData.

What's the problem? Should patch es to set PHASE?

[Thibault_Richard (Thibault)]

Data migration should occur when ECK downscales a cluster by removing one or more nodes. Before removing a node, the operator makes sure that there is no more data on it by updating the Cluster-level shard allocation and routing settings. This explains why you see that the node 'es-v7132-es-datas-2' is excluded from any allocation.

Depending on the amount of data to be moved, the migration can take hours. It is also possible that the migration will be blocked if your cluster cannot handle your data with the new smaller number of nodes (e.g.: an index with 1 primary and 2 replicas cannot be handled by a 2-nodes cluster).

What is the current and future topology of your cluster?
Can you share the output of GET /_cluster/health?level=indices&pretty?