# Unallocated shards when a node is removed **URL:** \ **Category:** Elasticsearch\ **Created:** [May 18, 2016, 11:41am UTC](https://discuss.elastic.co/t/unallocated-shards-when-a-node-is-removed/50331 "2016-05-18T11:41:05Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![vangap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangap/32/44927_2.png) [@vangap](https://discuss.elastic.co/u/vangap)\ **Post date:** [May 18, 2016, 11:41am UTC](https://discuss.elastic.co/t/unallocated-shards-when-a-node-is-removed/50331/1 "2016-05-18T11:41:05Z")
ES 2.1 AWS EC2, Private subnets. When ever I remove a node, shards that belong to this node don't get re balanced I think some times resetting the number of replicas solves the issue. Most of the times I end up restarting nodes. This is happening consistently every time. On the master node I see this > delaying recovery of [7bs2em65onshoizh][1] as it is not listed as assigned to target node {Impala}{HXQsJaPeTP2DTVY7WzA\_4Q} any ideas why this might be happening? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 18, 2016, 12:10pm UTC](https://discuss.elastic.co/t/unallocated-shards-when-a-node-is-removed/50331/2 "2016-05-18T12:10:46Z")
Do you have allocation disabled? Check `_cluster/settings` ---
**Author:** ![vangap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangap/32/44927_2.png) [@vangap](https://discuss.elastic.co/u/vangap)\ **Post date:** [May 18, 2016, 12:35pm UTC](https://discuss.elastic.co/t/unallocated-shards-when-a-node-is-removed/50331/3 "2016-05-18T12:35:13Z")
allocation is not disabled `_cluster/settings` shows `{"persistent":{},"transient":{}}` `_cat/shards` shows `UNASSIGNED` shards untill I restart one of the remaining nodes ---
**Author:** ![vangap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangap/32/44927_2.png) [@vangap](https://discuss.elastic.co/u/vangap)\ **Post date:** [May 18, 2016, 1:29pm UTC](https://discuss.elastic.co/t/unallocated-shards-when-a-node-is-removed/50331/4 "2016-05-18T13:29:58Z")
More info delayed\_timeout is default 1m there is no diskspace watermark limit on any of the remaining nodes. `_cluster/health` ```auto {"cluster_name":"abc","status":"yellow","timed_out":false,"number_of_nodes":2,"number_of_data_nodes":2,"active_primary_shards":222,"active_shards":296,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":148,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":66.66666666666666} ``` ---
**Author:** ![vangap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangap/32/44927_2.png) [@vangap](https://discuss.elastic.co/u/vangap)\ **Post date:** [May 19, 2016, 8:54am UTC](https://discuss.elastic.co/t/unallocated-shards-when-a-node-is-removed/50331/5 "2016-05-19T08:54:12Z")
Moving the discussion from [Add a notice note to README · Issue #1 · alicegoldfuss/shardnado · GitHub](https://github.com/alicegoldfuss/shardnado/issues/1#issuecomment-220253473) @s1monw I am not sure what this is > I think you should check the settings on your indices and cluster, use the explain feature on the \_reroute API and first find out why before you fix the why. 🙂 btw, I didn't use this shardnado tool, I was just saying that some times ES doesn't allocate shards on its own. our cluster is a basic one, doesn't have any routing values setup or anything. As posted in earlier comments in this thread, cluster settings are empty. I haven't disabled allocation. I checked on of the index settings also `/index-name/_settings`, nothing useful there about shards. I can reproduce this, happens every time I remove a node. please let me know how I can use this `_reroute` to debug this further. Thanks. ---
**Author:** ![s1monw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s1monw/32/3637_2.png) [@s1monw](https://discuss.elastic.co/u/s1monw)\ **Post date:** [May 19, 2016, 9:10am UTC](https://discuss.elastic.co/t/unallocated-shards-when-a-node-is-removed/50331/6 "2016-05-19T09:10:45Z")
hey [the \_reroute API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-reroute.html) allows you to manually reroute shards to a node. you can use a parameter called `explain=true` that would give you the reasons why this allocation could or could not be applied. That should tell you why shards are not allocated and / or if they are throttled. If you call that API with an empty body you can trigger a new round of rerouting and get some information about all the shards. That should give you a much better idea of what is going on. If you can paste that output here I can take a look. Also send me the index settings of the index that is not allocating. simon ---
**Author:** ![vangap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangap/32/44927_2.png) [@vangap](https://discuss.elastic.co/u/vangap)\ **Post date:** [May 19, 2016, 9:18am UTC](https://discuss.elastic.co/t/unallocated-shards-when-a-node-is-removed/50331/7 "2016-05-19T09:18:47Z")
Ok, so I have run this command ```auto curl -XPOST 'localhost:9200/_cluster/reroute?pretty&explain' -d '{ "commands" : [ { "allocate" : { "index" : "1qp6axstrjub7ouw", "shard" : 1, "node" : "Nimrod" } } ] }' ``` this allocated the replica 1 to the node Nimrod which was unassigned before. Remaining all unassigned shards also got allocated after this. here is the output of that command in `dry_run` mode if that is of any use [https://gist.github.com/vanga/146c356c20758765afd22c4c739df572](https://gist.github.com/vanga/146c356c20758765afd22c4c739df572) ---
**Author:** ![s1monw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s1monw/32/3637_2.png) [@s1monw](https://discuss.elastic.co/u/s1monw)\ **Post date:** [May 19, 2016, 9:41am UTC](https://discuss.elastic.co/t/unallocated-shards-when-a-node-is-removed/50331/8 "2016-05-19T09:41:27Z")
Can I see the index settings of that index 1qp6axstrjub7ouw ---
**Author:** ![s1monw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s1monw/32/3637_2.png) [@s1monw](https://discuss.elastic.co/u/s1monw)\ **Post date:** [May 19, 2016, 9:47am UTC](https://discuss.elastic.co/t/unallocated-shards-when-a-node-is-removed/50331/9 "2016-05-19T09:47:43Z")
> here is the output of that command in dry\_run mode if that is of any use I think this is a bug in delayed allocation that misses to kick off another round of shard allocation. is there a chance for you to upgrade to 2.3 at some point? Also you can simulate that missing round of allocaiotn by calling `_reroute` with an empty body. ---
**Author:** ![vangap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangap/32/44927_2.png) [@vangap](https://discuss.elastic.co/u/vangap)\ **Post date:** [May 19, 2016, 9:51am UTC](https://discuss.elastic.co/t/unallocated-shards-when-a-node-is-removed/50331/10 "2016-05-19T09:51:06Z")
here is the output of cluster settings ```auto [root@ip-172-31-48-137 ~]# curl localhost:9200/1qp6axstrjub7ouw/_settings?pretty { "1qp6axstrjub7ouw" : { "settings" : { "index" : { "creation_date" : "1446015175417", "number_of_shards" : "5", "number_of_replicas" : "1", "uuid" : "gOZtw-IGS2ap2Cz6tJt4MA", "version" : { "created" : "2000051" } } } } } ``` We will eventually move to 2.3 (may be in few weeks or 1-2 months not sure), I saw there are some breaking changes from 2.1-2.2, need to look into them and plan. ---
**Author:** ![s1monw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s1monw/32/3637_2.png) [@s1monw](https://discuss.elastic.co/u/s1monw)\ **Post date:** [May 19, 2016, 9:52am UTC](https://discuss.elastic.co/t/unallocated-shards-when-a-node-is-removed/50331/11 "2016-05-19T09:52:47Z")
ok so can you provoke this problem again and see if an empty reroute fixes it? ---
**Author:** ![vangap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangap/32/44927_2.png) [@vangap](https://discuss.elastic.co/u/vangap)\ **Post date:** [May 19, 2016, 10:01am UTC](https://discuss.elastic.co/t/unallocated-shards-when-a-node-is-removed/50331/12 "2016-05-19T10:01:10Z")
Ok `curl -XPOST 'localhost:9200/_cluster/reroute'` this does allocate those unassigned ones ---
**Author:** ![s1monw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s1monw/32/3637_2.png) [@s1monw](https://discuss.elastic.co/u/s1monw)\ **Post date:** [May 19, 2016, 12:58pm UTC](https://discuss.elastic.co/t/unallocated-shards-when-a-node-is-removed/50331/13 "2016-05-19T12:58:20Z")
perfect, I think you ran into one of those bugs where delayed allocation missed a reroute. Can you please upgrade to the latest and see if the bug persists? If so please open an issue on our issue tracker! thanks! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:50pm UTC](https://discuss.elastic.co/t/unallocated-shards-when-a-node-is-removed/50331/14 "2017-07-05T22:50:25Z")