# Not recovering all shards, why? **URL:** \ **Category:** Elasticsearch\ **Created:** [April 20, 2011, 9:53am UTC](https://discuss.elastic.co/t/not-recovering-all-shards-why/4264 "2011-04-20T09:53:56Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [April 20, 2011, 9:53am UTC](https://discuss.elastic.co/t/not-recovering-all-shards-why/4264/1 "2011-04-20T09:53:56Z")
Hi, I see strange thing with the latest master. (The below log output can be found also here [https://gist.github.com/930879](https://gist.github.com/930879)) I create a new index (lets call it a "dummyIndex"), 3 shards, 1 replica. Because the cluster has only one node then the status is always yellow, that is fine. [2011-04-20 05:28:41,658][INFO][cluster.metadata] [Hellstrom, Patsy] [dummyIndex] creating index, cause [auto(index api)], shards [3]/[1], mappings [mail] [2011-04-20 05:28:41,688][DEBUG][index.gateway] [Hellstrom, Patsy] [dummyIndex][0] starting recovery from local ... [2011-04-20 05:28:41,693][DEBUG][index.gateway] [Hellstrom, Patsy] [dummyIndex][0] recovery completed from local, took [6ms] index : files [0] with total\_size [0b], took[0s] : recovered\_files [0] with total\_size [0b] : reusing\_files [0] with total\_size [0b] translog : number\_of\_operations [0], took [5ms] [2011-04-20 05:28:41,713][DEBUG][index.gateway] [Hellstrom, Patsy] [dummyIndex][1] starting recovery from local ... [2011-04-20 05:28:41,719][DEBUG][index.gateway] [Hellstrom, Patsy] [dummyIndex][1] recovery completed from local, took [6ms] index : files [0] with total\_size [0b], took[0s] : recovered\_files [0] with total\_size [0b] : reusing\_files [0] with total\_size [0b] translog : number\_of\_operations [0], took [5ms] [2011-04-20 05:28:41,739][DEBUG][index.gateway] [Hellstrom, Patsy] [dummyIndex][2] starting recovery from local ... [2011-04-20 05:28:41,744][DEBUG][index.gateway] [Hellstrom, Patsy] [dummyIndex][2] recovery completed from local, took [5ms] index : files [0] with total\_size [0b], took[0s] : recovered\_files [0] with total\_size [0b] : reusing\_files [0] with total\_size [0b] translog : number\_of\_operations [0], took [4ms] [2011-04-20 05:28:41,821][INFO][cluster.metadata] [Hellstrom, Patsy] [dummyIndex] update\_mapping [mail] (dynamic) Then I index some data into it and shutdown the cluster (node is using the default local gateway). Later I start it again but one shard is not recovered, why? [2011-04-20 05:37:27,050][DEBUG][gateway.local] [Vashti] [dummyIndex][0]: throttling allocation [[dummyIndex][0], node[null], [P], s[UNASSIGNED]] to [[Vashti][X0Hk2SAVTfOBjVUcMAPOCQ][inet[/10.16.88.223:9300]]] on primary allocation [2011-04-20 05:37:27,051][DEBUG][gateway.local] [Vashti] [dummyIndex][1]: throttling allocation [[dummyIndex][1], node[null], [P], s[UNASSIGNED]] to [[Vashti][X0Hk2SAVTfOBjVUcMAPOCQ][inet[/10.16.88.223:9300]]] on primary allocation [2011-04-20 05:37:27,051][DEBUG][gateway.local] [Vashti] [dummyIndex][2]: not allocating, number\_of\_allocated\_shards\_found [0], required\_number [1] As a side note, there are more indices created the same way on this node (say ~8 other indices) and they are always correctly recovered, only this specific index is not. May this has to do something with the fact that when I am indexing data into the cluster then some threads on my client are hanging (not sure if these threads are indexing into dummyIndex, will try to find out), however, even if the threads are hanging I see the status on all indices are yellow, not red, before the shudown. Regards, Lukas ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 20, 2011, 10:11am UTC](https://discuss.elastic.co/t/not-recovering-all-shards-why/4264/2 "2011-04-20T10:11:14Z")
If the status is yellow, then those indices should be recovered. What we can see is that there is a missing shard so its not being recovered. Are you sure that it was yellow before you shut it down? On Wednesday, April 20, 2011 at 12:53 PM, Lukáš Vlček wrote: > Hi, > > I see strange thing with the latest master. (The below log output can be found also here [https://gist.github.com/930879](https://gist.github.com/930879)) > > I create a new index (lets call it a "dummyIndex"), 3 shards, 1 replica. Because the cluster has only one node then the status is always yellow, that is fine. > > [2011-04-20 05:28:41,658][INFO][cluster.metadata] [Hellstrom, Patsy] [dummyIndex] creating index, cause [auto(index api)], shards [3]/[1], mappings [mail] > [2011-04-20 05:28:41,688][DEBUG][index.gateway] [Hellstrom, Patsy] [dummyIndex][0] starting recovery from local ... > [2011-04-20 05:28:41,693][DEBUG][index.gateway] [Hellstrom, Patsy] [dummyIndex][0] recovery completed from local, took [6ms] > index : files [0] with total\_size [0b], took[0s] > : recovered\_files [0] with total\_size [0b] > : reusing\_files [0] with total\_size [0b] > translog : number\_of\_operations [0], took [5ms] > [2011-04-20 05:28:41,713][DEBUG][index.gateway] [Hellstrom, Patsy] [dummyIndex][1] starting recovery from local ... > [2011-04-20 05:28:41,719][DEBUG][index.gateway] [Hellstrom, Patsy] [dummyIndex][1] recovery completed from local, took [6ms] > index : files [0] with total\_size [0b], took[0s] > : recovered\_files [0] with total\_size [0b] > : reusing\_files [0] with total\_size [0b] > translog : number\_of\_operations [0], took [5ms] > [2011-04-20 05:28:41,739][DEBUG][index.gateway] [Hellstrom, Patsy] [dummyIndex][2] starting recovery from local ... > [2011-04-20 05:28:41,744][DEBUG][index.gateway] [Hellstrom, Patsy] [dummyIndex][2] recovery completed from local, took [5ms] > index : files [0] with total\_size [0b], took[0s] > : recovered\_files [0] with total\_size [0b] > : reusing\_files [0] with total\_size [0b] > translog : number\_of\_operations [0], took [4ms] > [2011-04-20 05:28:41,821][INFO][cluster.metadata] [Hellstrom, Patsy] [dummyIndex] update\_mapping [mail] (dynamic) > > Then I index some data into it and shutdown the cluster (node is using the default local gateway). > Later I start it again but one shard is not recovered, why? > > [2011-04-20 05:37:27,050][DEBUG][gateway.local] [Vashti] [dummyIndex][0]: throttling allocation [[dummyIndex][0], node[null], [P], s[UNASSIGNED]] to [[Vashti][X0Hk2SAVTfOBjVUcMAPOCQ][inet[/10.16.88.223:9300]]] on primary allocation > [2011-04-20 05:37:27,051][DEBUG][gateway.local] [Vashti] [dummyIndex][1]: throttling allocation [[dummyIndex][1], node[null], [P], s[UNASSIGNED]] to [[Vashti][X0Hk2SAVTfOBjVUcMAPOCQ][inet[/10.16.88.223:9300]]] on primary allocation > [2011-04-20 05:37:27,051][DEBUG][gateway.local] [Vashti] [dummyIndex][2]: not allocating, number\_of\_allocated\_shards\_found [0], required\_number [1] > > As a side note, there are more indices created the same way on this node (say ~8 other indices) and they are always correctly recovered, only this specific index is not. May this has to do something with the fact that when I am indexing data into the cluster then some threads on my client are hanging (not sure if these threads are indexing into dummyIndex, will try to find out), however, even if the threads are hanging I see the status on all indices are yellow, not red, before the shudown. > > Regards, > Lukas ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [April 20, 2011, 10:19am UTC](https://discuss.elastic.co/t/not-recovering-all-shards-why/4264/3 "2011-04-20T10:19:22Z")
Absolutely, it was yellow for all shards. Strange... I can simulate this on my local machine but probably it would be very hard to provide recreation script ☹ On Wed, Apr 20, 2011 at 12:11 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > If the status is yellow, then those indices should be recovered. What we > can see is that there is a missing shard so its not being recovered. Are you > sure that it was yellow before you shut it down? > > On Wednesday, April 20, 2011 at 12:53 PM, Lukáš Vlček wrote: > > Hi, > > I see strange thing with the latest master. (The below log output can be > found also here [https://gist.github.com/930879](https://gist.github.com/930879)) > > I create a new index (lets call it a "dummyIndex"), 3 shards, 1 replica. > Because the cluster has only one node then the status is always yellow, that > is fine. > > [2011-04-20 05:28:41,658][INFO][cluster.metadata] [Hellstrom, > Patsy] [dummyIndex] creating index, cause [auto(index api)], shards [3]/[1], > mappings [mail] > [2011-04-20 05:28:41,688][DEBUG][index.gateway] [Hellstrom, > Patsy] [dummyIndex][0] starting recovery from local ... > [2011-04-20 05:28:41,693][DEBUG][index.gateway] [Hellstrom, > Patsy] [dummyIndex][0] recovery completed from local, took [6ms] > index : files [0] with total\_size [0b], took[0s] > : recovered\_files [0] with total\_size [0b] > : reusing\_files [0] with total\_size [0b] > translog : number\_of\_operations [0], took [5ms] > [2011-04-20 05:28:41,713][DEBUG][index.gateway] [Hellstrom, > Patsy] [dummyIndex][1] starting recovery from local ... > [2011-04-20 05:28:41,719][DEBUG][index.gateway] [Hellstrom, > Patsy] [dummyIndex][1] recovery completed from local, took [6ms] > index : files [0] with total\_size [0b], took[0s] > : recovered\_files [0] with total\_size [0b] > : reusing\_files [0] with total\_size [0b] > translog : number\_of\_operations [0], took [5ms] > [2011-04-20 05:28:41,739][DEBUG][index.gateway] [Hellstrom, > Patsy] [dummyIndex][2] starting recovery from local ... > [2011-04-20 05:28:41,744][DEBUG][index.gateway] [Hellstrom, > Patsy] [dummyIndex][2] recovery completed from local, took [5ms] > index : files [0] with total\_size [0b], took[0s] > : recovered\_files [0] with total\_size [0b] > : reusing\_files [0] with total\_size [0b] > translog : number\_of\_operations [0], took [4ms] > [2011-04-20 05:28:41,821][INFO][cluster.metadata] [Hellstrom, > Patsy] [dummyIndex] update\_mapping [mail] (dynamic) > > Then I index some data into it and shutdown the cluster (node is using the > default local gateway). > Later I start it again but one shard is not recovered, why? > > [2011-04-20 05:37:27,050][DEBUG][gateway.local] [Vashti] > [dummyIndex][0]: throttling allocation [[dummyIndex][0], node[null], [P], > s[UNASSIGNED]] to [[Vashti][X0Hk2SAVTfOBjVUcMAPOCQ][inet[/10.16.88.223:9300]]] > on primary allocation > [2011-04-20 05:37:27,051][DEBUG][gateway.local] [Vashti] > [dummyIndex][1]: throttling allocation [[dummyIndex][1], node[null], [P], > s[UNASSIGNED]] to [[Vashti][X0Hk2SAVTfOBjVUcMAPOCQ][inet[/10.16.88.223:9300]]] > on primary allocation > [2011-04-20 05:37:27,051][DEBUG][gateway.local] [Vashti] > [dummyIndex][2]: not allocating, number\_of\_allocated\_shards\_found [0], > required\_number [1] > > As a side note, there are more indices created the same way on this node > (say ~8 other indices) and they are always correctly recovered, only this > specific index is not. May this has to do something with the fact that when > I am indexing data into the cluster then some threads on my client are > hanging (not sure if these threads are indexing into dummyIndex, will try to > find out), however, even if the threads are hanging I see the status on all > indices are yellow, not red, before the shudown. > > Regards, > Lukas ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:08am UTC](https://discuss.elastic.co/t/not-recovering-all-shards-why/4264/4 "2017-07-06T04:08:00Z")