I tried to move indices from one node to another using reroute API but I had the following error.
reason: "can't move from node-1 to node-2, , since its not allowed, reason: .. [NO(the node is above the low watermark cluster setting [cluster.routing.allocation.disk.watermark.low=85%], using more disk space than the maximum allowed [85.0%], actual free: [10.485435061984592%])][YES(below shard recovery limit of outgoing: [0 < 2] incoming: [0 < 2])]
Why is it not allowed? my node-1 has 90% disk usage while my node-2 is at 27%.
Sorry, I think it was my error. I misplaced the "from" and "to" node.
My question though, is it not possible to move the shards when it already has a replica?
Why only one node is almost full when almost all indices in the hot phase have replica?
Do you have any suggestions on how to free up the disk in my master node beside moving indices?
Sorry, I don't quite understand what removing the replica will do.
I have checked the replica size and index size. I found that they are the same size. So I think removing the replica, moving the primary shard to another node then making another replica won't change the current status, right?
But if the primary and shard have the same size and most of the indices have replicas, why the disk usage is different between the hot nodes?
I delete some old indices on the warm node yesterday, and voila, the disk usage in a hot node is also decreasing
I mean, even if I temporarily remove them, let them get balanced and then added them back as you have suggested, isn't it basically will have the same amount of disk usage since the shard and replica have exact same size?
Sorry, I think I misunderstood.
At first, I thought it will gonna help reduce the disk difference between 2 nodes.
Because I don't want to rebalance it, since balanced shards don't mean balanced disk usage, right?
So then my last option is to reduce the time the data is stored in the hot phase by rollover the indices early then move them to warm tier?
Ok, then if you remove the replicas on the hot nodes then Elasticsearch will first try to balance the shard count so it's the same, and will factor in disk use of each node as it does so. You may not end up with 100% matching numbers, but it should be better than what you have now.