# Reroute API **URL:** \ **Category:** Elasticsearch\ **Created:** [October 18, 2022, 12:20am UTC](https://discuss.elastic.co/t/reroute-api/316832 "2022-10-18T00:20:13Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![wijamw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wijamw/32/111240_2.png) [@wijamw](https://discuss.elastic.co/u/wijamw)\ **Post date:** [October 18, 2022, 12:20am UTC](https://discuss.elastic.co/t/reroute-api/316832/1 "2022-10-18T00:20:14Z")
I tried to move indices from one node to another using reroute API but I had the following error. ```auto type: illegal_argument_exception reason: "can't move from node-1 to node-2, , since its not allowed, reason: .. [NO(the node is above the low watermark cluster setting [cluster.routing.allocation.disk.watermark.low=85%], using more disk space than the maximum allowed [85.0%], actual free: [10.485435061984592%])][YES(below shard recovery limit of outgoing: [0 < 2] incoming: [0 < 2])] ``` Why is it not allowed? my node-1 has 90% disk usage while my node-2 is at 27%. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 18, 2022, 12:25am UTC](https://discuss.elastic.co/t/reroute-api/316832/2 "2022-10-18T00:25:21Z")
What's the output from `GET /_cat/nodes?v`? ---
**Author:** ![wijamw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wijamw/32/111240_2.png) [@wijamw](https://discuss.elastic.co/u/wijamw)\ **Post date:** [October 18, 2022, 12:50am UTC](https://discuss.elastic.co/t/reroute-api/316832/5 "2022-10-18T00:50:23Z")
![get node](https://us1.discourse-cdn.com/elastic/original/3X/1/d/1d57b23e69af3ac4401769205a6fdf6000c05268.png) ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 18, 2022, 12:51am UTC](https://discuss.elastic.co/t/reroute-api/316832/6 "2022-10-18T00:51:18Z")
No worries, you can use the code formatter you used above and it'll be readable 🙂 > [@wijamw](#): > > `can't move from node-1 to node-2` That doesn't match the names in your output? ---
**Author:** ![wijamw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wijamw/32/111240_2.png) [@wijamw](https://discuss.elastic.co/u/wijamw)\ **Post date:** [October 18, 2022, 1:13am UTC](https://discuss.elastic.co/t/reroute-api/316832/7 "2022-10-18T01:13:38Z")
Sorry, I think it was my error. I misplaced the "from" and "to" node. My bad. My question though, is it not possible to move the shards when it already has a replica? Why only one node is almost full when almost all indices in the hot phase have replica? Do you have any suggestions on how to free up the disk in my master node beside moving indices? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 18, 2022, 1:22am UTC](https://discuss.elastic.co/t/reroute-api/316832/8 "2022-10-18T01:22:47Z")
> [@wijamw](#): > > My question though, is it not possible to move the shards when it already has a replica? You cannot allocate a replica on the same node as the primary, no. > [@wijamw](#): > > Why only one node is almost full when almost all indices in the hot phase have replica? We'd need to see `_cat/allocation?v` to comment. ---
**Author:** ![wijamw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wijamw/32/111240_2.png) [@wijamw](https://discuss.elastic.co/u/wijamw)\ **Post date:** [October 18, 2022, 1:33am UTC](https://discuss.elastic.co/t/reroute-api/316832/9 "2022-10-18T01:33:24Z")
> [@warkolm](#): > > We'd need to see `_cat/allocation?v` to comment. ```auto shards disk.indices disk.used disk.avail disk.total disk.percent node 365 158.4gb 163.4gb 16.5gb 180gb 90 instance-0000000001 326 43.1gb 50.1gb 129.8gb 180gb 27 instance-0000000000 97 165.7gb 165.7gb 214.2gb 380gb 43 instance-0000000004 ``` Basically what I want to do earlier is to move some indices from instance-01 to instance-00 to make the disk usage not too imbalanced. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 18, 2022, 2:31am UTC](https://discuss.elastic.co/t/reroute-api/316832/10 "2022-10-18T02:31:06Z")
Are you using Elastic Cloud? Are you using hot/cold tiering or other allocation filtering? ---
**Author:** ![wijamw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wijamw/32/111240_2.png) [@wijamw](https://discuss.elastic.co/u/wijamw)\ **Post date:** [October 18, 2022, 2:42am UTC](https://discuss.elastic.co/t/reroute-api/316832/11 "2022-10-18T02:42:08Z")
Yes I am using elastic cloud, I am using 2 instance for hot and 1 for 1 warm ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 18, 2022, 2:48am UTC](https://discuss.elastic.co/t/reroute-api/316832/12 "2022-10-18T02:48:57Z")
OK thanks for clarifying! What version are you running? I would try temporarily removing replicas from a few indices and then seeing if allocation balances out, then add them back. ---
**Author:** ![wijamw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wijamw/32/111240_2.png) [@wijamw](https://discuss.elastic.co/u/wijamw)\ **Post date:** [October 18, 2022, 3:06am UTC](https://discuss.elastic.co/t/reroute-api/316832/13 "2022-10-18T03:06:21Z")
> [@warkolm](#): > > What version are you running? I am running 7.17 > [@warkolm](#): > > I would try temporarily removing replicas from a few indices and then seeing if allocation balances out, then add them back. Could you give me some references on how to do that? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 18, 2022, 4:21am UTC](https://discuss.elastic.co/t/reroute-api/316832/14 "2022-10-18T04:21:39Z")
`PUT indexname/_settings { "index" : { "number_of_replicas" : 0 } }'` will do it. You can use wildcards too ---
**Author:** ![wijamw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wijamw/32/111240_2.png) [@wijamw](https://discuss.elastic.co/u/wijamw)\ **Post date:** [October 20, 2022, 11:33pm UTC](https://discuss.elastic.co/t/reroute-api/316832/15 "2022-10-20T23:33:42Z")
Sorry, I don't quite understand what removing the replica will do. I have checked the replica size and index size. I found that they are the same size. So I think removing the replica, moving the primary shard to another node then making another replica won't change the current status, right? But if the primary and shard have the same size and most of the indices have replicas, why the disk usage is different between the hot nodes? I delete some old indices on the warm node yesterday, and voila, the disk usage in a hot node is also decreasing 😅 ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 21, 2022, 1:35am UTC](https://discuss.elastic.co/t/reroute-api/316832/16 "2022-10-21T01:35:45Z")
> [@wijamw](#): > > Sorry, I don't quite understand what removing the replica will do. It removes replica copy of the data, so you will only have your primary shards. Then you can move them. ---
**Author:** ![wijamw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wijamw/32/111240_2.png) [@wijamw](https://discuss.elastic.co/u/wijamw)\ **Post date:** [October 31, 2022, 2:00am UTC](https://discuss.elastic.co/t/reroute-api/316832/17 "2022-10-31T02:00:22Z")
I mean, even if I temporarily remove them, let them get balanced and then added them back as you have suggested, isn't it basically will have the same amount of disk usage since the shard and replica have exact same size? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 31, 2022, 2:12am UTC](https://discuss.elastic.co/t/reroute-api/316832/18 "2022-10-31T02:12:32Z")
Yes. However if you want to balance shard counts between your hot nodes then removing replicas will trigger Elasticsearch to do that. Then you can add replicas back in. ---
**Author:** ![wijamw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wijamw/32/111240_2.png) [@wijamw](https://discuss.elastic.co/u/wijamw)\ **Post date:** [October 31, 2022, 4:57am UTC](https://discuss.elastic.co/t/reroute-api/316832/19 "2022-10-31T04:57:26Z")
Sorry, I think I misunderstood. At first, I thought it will gonna help reduce the disk difference between 2 nodes. Because I don't want to rebalance it, since balanced shards don't mean balanced disk usage, right? So then my last option is to reduce the time the data is stored in the hot phase by rollover the indices early then move them to warm tier? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 31, 2022, 5:50am UTC](https://discuss.elastic.co/t/reroute-api/316832/20 "2022-10-31T05:50:33Z")
If you're talking about balancing between hot and warm, then the only way to do that is to reduce the time your hot shards stay on that tier. ---
**Author:** ![wijamw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wijamw/32/111240_2.png) [@wijamw](https://discuss.elastic.co/u/wijamw)\ **Post date:** [October 31, 2022, 1:09pm UTC](https://discuss.elastic.co/t/reroute-api/316832/21 "2022-10-31T13:09:20Z")
No, I am talking about balancing between 2 hot nodes ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 31, 2022, 11:39pm UTC](https://discuss.elastic.co/t/reroute-api/316832/22 "2022-10-31T23:39:27Z")
Ok, then if you remove the replicas on the hot nodes then Elasticsearch will first try to balance the shard count so it's the same, and will factor in disk use of each node as it does so. You may not end up with 100% matching numbers, but it should be better than what you have now. [Next page](https://discuss.elastic.co/t/reroute-api/316832.md?page=2)