# Allocation/re-routing of unassigned shards **URL:** \ **Category:** Elasticsearch\ **Created:** [November 13, 2017, 9:41pm UTC](https://discuss.elastic.co/t/allocation-re-routing-of-unassigned-shards/107466 "2017-11-13T21:41:53Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Nikolay\_Petrov](https://avatars.discourse-cdn.com/v4/letter/n/4af34b/32.png) [@Nikolay\_Petrov](https://discuss.elastic.co/u/Nikolay_Petrov)\ **Post date:** [November 13, 2017, 9:41pm UTC](https://discuss.elastic.co/t/allocation-re-routing-of-unassigned-shards/107466/1 "2017-11-13T21:41:53Z")
Hi, could you please help to solve ongoing issue with unassigned shards? The story of these sad example is that cluster with one node goes down(terminated),disk with a elastic data attached to another system with the same 'path data', and now i'm trying to ignore previous node and relocate the outstanding shards to the new one cluster new cluster info: > ip heap.percent ram.percent cpu load\_1m load\_5m load\_15m node.role master name > 10.51.184.252 47 76 19 5.03 4.85 3.09 mdi \* cae1elk-000-001 > 10.51.184.101 51 61 15 0.62 0.96 1.17 mdi - cae1elk-000-002 overall index information: > [root@cae1elk-000-001 ~]# curl -s -XGET http://`hostname`:9200/\_cat/indices?v | grep ^green | wc -l > 15 > [root@cae1elk-000-001 ~]# curl -s -XGET http://`hostname`:9200/\_cat/indices?v | grep ^red | wc -l > 190 cluster status: > { > "cluster\_name" : "cae\_ops\_cluster", > "status" : "red", > "timed\_out" : false, > "number\_of\_nodes" : 2, > "number\_of\_data\_nodes" : 2, > "active\_primary\_shards" : 218, > "active\_shards" : 248, > "relocating\_shards" : 0, > "initializing\_shards" : 2, > "unassigned\_shards" : 568, > "delayed\_unassigned\_shards" : 0, > "number\_of\_pending\_tasks" : 0, > "number\_of\_in\_flight\_fetch" : 0, > "task\_max\_waiting\_in\_queue\_millis" : 0, > "active\_shards\_percent\_as\_number" : 30.317848410757946 > } when i'm applying workaround from: [https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-reroute.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-reroute.html) i'm getting the: > curl -s -XGET http://`hostname`:9200/\_cluster/allocation/explain?pretty > { > "index" : "snmptrap-events-2017.05.17", > "shard" : 0, > "primary" : true, > "current\_state" : "unassigned", > "unassigned\_info" : { > "reason" : "CLUSTER\_RECOVERED", > "at" : "2017-11-13T14:36:23.256Z", > "last\_allocation\_status" : "no\_valid\_shard\_copy" > }, > "can\_allocate" : "no\_valid\_shard\_copy", > "allocate\_explanation" : "cannot allocate because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster", > "node\_allocation\_decisions" : [ > { > "node\_id" : "DJtotx\_7SMaGCUecvyknEQ", > "node\_name" : "cae1elk-000-001", > "transport\_address" : "10.51.184.252:9300", > "node\_attributes" : { > "aws\_availability\_zone" : "us-east-1a" > }, > "node\_decision" : "no", > "store" : { > "found" : false > } > }, > { > "node\_id" : "jcNMPApaSCeptag3PxW\_8w", > "node\_name" : "cae1elk-000-002", > "transport\_address" : "10.51.184.101:9300", > "node\_attributes" : { > "aws\_availability\_zone" : "us-east-1a" > }, > "node\_decision" : "no", > "store" : { > "found" : false > } > } > ] > } > [root@cae1elk-000-001 ~]# curl -XPOST http://`hostname`:9200/\_cluster/reroute -d '@4' > {"error":{"root\_cause":[{"type":"illegal\_argument\_exception","reason":"[move\_allocation] can't move 0, failed to find it on node {cae1elk-000-001}{DJtotx\_7SMaGCUecvyknEQ}{50-UmnMpQWO2CB2t2R9wHQ}{10.51.184.252}{10.51.184.252:9300}{aws\_availability\_zone=us-east-1a}"}],"type":"illegal\_argument\_exception","reason":"[move\_allocation] can't move 0, failed to find it on node {cae1elk-000-001}{DJtotx\_7SMaGCUecvyknEQ}{50-UmnMpQWO2CB2t2R9wHQ}{10.51.184.252}{10.51.184.252:9300}{aws\_availability\_zone=us-east-1a}"},"status":400}[root@cae1elk-000-001 ~]# cat 4 > { > "commands" : [ > { > "move" : { > "index" : "snmptrap-events-2017.05.17", "shard" : 0, > "from\_node" : "cae1elk-000-001", "to\_node" : "cae1elk-000-002" > } > }, > { > "allocate\_replica" : { > "index" : "snmptrap-events-2017.05.17", "shard" : 1, > "node" : "cae1elk-000-002" > } > } > ] > } Is their any possibility in my case somehow to restore the shards, even with lost info? ---
**Author:** ![varun\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/varun_kumar/32/24170_2.png) [@varun\_kumar](https://discuss.elastic.co/u/varun_kumar)\ **Post date:** [November 14, 2017, 3:07am UTC](https://discuss.elastic.co/t/allocation-re-routing-of-unassigned-shards/107466/2 "2017-11-14T03:07:02Z")
What was the replication factor? We faced this issue with Elastic 2.4 cluster, where some nodes had root volume full and as a result, ES started acting out. Consequently, new shards got corrupted both primary and few of replica. First thing we did was disabled reallocation to prevent further moving of data. Tried to identify the nodes with good replicas. Deleted a node from the cluster, still having re-allocation disabled. Reattached the EBS volume to a fresh node and then enabled reacclocation. Several of the index became green, however there was one which was still red with 2 shards being bad. After a lot of try, we gave up and restored a fresh instance of ES cluster from last night's backup. And copied the data on the bad index across cluster. Would be watching out the thread if there is a better solution with ES 5x. Good luck! ---
**Author:** ![Nikolay\_Petrov](https://avatars.discourse-cdn.com/v4/letter/n/4af34b/32.png) [@Nikolay\_Petrov](https://discuss.elastic.co/u/Nikolay_Petrov)\ **Post date:** [November 14, 2017, 11:29am UTC](https://discuss.elastic.co/t/allocation-re-routing-of-unassigned-shards/107466/3 "2017-11-14T11:29:30Z")
Fixed somehow,but still trying to investigate,what kind of workaround was working in my case. Thanks anyway for suggested way! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 12, 2017, 11:29am UTC](https://discuss.elastic.co/t/allocation-re-routing-of-unassigned-shards/107466/4 "2017-12-12T11:29:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.