# Shard getting "stuck" during a rebalance **URL:** \ **Category:** Elasticsearch\ **Created:** [October 22, 2012, 9:20am UTC](https://discuss.elastic.co/t/shard-getting-stuck-during-a-rebalance/9435 "2012-10-22T09:20:15Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Jamie\_Matthews](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jamie_matthews/32/2669_2.png) [@Jamie\_Matthews](https://discuss.elastic.co/u/Jamie_Matthews)\ **Post date:** [October 22, 2012, 9:20am UTC](https://discuss.elastic.co/t/shard-getting-stuck-during-a-rebalance/9435/1 "2012-10-22T09:20:15Z")
Hi, We have a problem with a node getting "stuck" during a rebalance. It's a five-node cluster running ES 0.19.1 with a single large index split into five shards, with a replication level of 2. One of the shards has been in INITIALIZING state since yesterday, and the logs on that node show org.elasticsearch.index.IndexShardMissingException. Screenshot of the HEAD interface here: [http://imgur.com/T7I4r](http://imgur.com/T7I4r) - the "stuck" shard is number 1 on the fifth node in the list. My concern is that this shard now has only one "good" copy, and we are concerned about losing the index if anything goes wrong with it (for various reasons it's currently very difficult for us to reindex, this is being worked on). Will just stopping and restarting that node fix the issue? Is this likely to cause any further problems? Many thanks, Jamie -- ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [October 22, 2012, 10:14am UTC](https://discuss.elastic.co/t/shard-getting-stuck-during-a-rebalance/9435/2 "2012-10-22T10:14:15Z")
Hello Jamie, On Mon, Oct 22, 2012 at 12:20 PM, Jamie Matthews [jamie.matthews@gmail.com](mailto:jamie.matthews@gmail.com) wrote: > Hi, > > We have a problem with a node getting "stuck" during a rebalance. It's a > five-node cluster running ES 0.19.1 with a single large index split into > five shards, with a replication level of 2. One of the shards has been in > INITIALIZING state since yesterday, and the logs on that node show > org.elasticsearch.index.IndexShardMissingException. If you use Local Gateway, which is the default, this issue is most likely caused by either something being corrupted on the filesystem, or, if you recently restarted your cluster, some inappropriate recovery settings here: > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... With the default recovery settings and 4 nodes, if you restart your whole cluster, all nodes might try to recover their shards from the local gateway at the same time. This might lead to inconsistencies. > Screenshot of the HEAD interface here: [Imgur: The magic of the Internet](http://imgur.com/T7I4r) - the "stuck" > shard is number 1 on the fifth node in the list. > > My concern is that this shard now has only one "good" copy, and we are > concerned about losing the index if anything goes wrong with it (for various > reasons it's currently very difficult for us to reindex, this is being > worked on). > > Will just stopping and restarting that node fix the issue? Is this likely to > cause any further problems? AFAIK, restarting a node in your situation will make the contents from other nodes to be replicated to the restarted node. Since your other nodes have "good" shards, the problem should get fixed. The worst thing that could happen(tm) is that more/all shards on that node to get faulty following the restart. Which isn't that much different from the situation you're in now. You still have to find a solution to what would basically be the same problem. ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:07am UTC](https://discuss.elastic.co/t/shard-getting-stuck-during-a-rebalance/9435/3 "2017-07-06T03:07:36Z")
Closed on July 6, 2017, 3:07am UTC