# Non-recovering index shard **URL:** \ **Category:** Elasticsearch\ **Created:** [August 10, 2011, 5:36pm UTC](https://discuss.elastic.co/t/non-recovering-index-shard/5117 "2011-08-10T17:36:49Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Pat\_Christopher](https://avatars.discourse-cdn.com/v4/letter/p/f14d63/32.png) [@Pat\_Christopher](https://discuss.elastic.co/u/Pat_Christopher)\ **Post date:** [August 10, 2011, 5:36pm UTC](https://discuss.elastic.co/t/non-recovering-index-shard/5117/1 "2011-08-10T17:36:49Z")
Hey guys, After Mondays AWS-EC2 failbasket I ended up with splitbrain and a number of rather strange configurations. I've cleaned out those with a judicious use of kill and restarting the ES nodes. My index has 5 shards and one replica on six data nodes. After my kill fiesta all five shards were yellow or red. Four of them have come back to green and are accepting writes again. The fifth shard has stubbornly remained at yellow even after closing and opening the index. It claims to have one active shard and one initializing shard. Its been initializing for about 20 hours now and I don't think its going to finish. When the other shards were initializing there was a tremendous amount of disk activity, now there is nothing spectacular going on. 1. how can I kick the last initializing shard to work? will an increase in shards cause it to rebalance and fix itself or will that only cause more problems? 2. two of the six data nodes have no data on them. I'm not entirely what they're doing but I'd like to get them involved. Any suggestions on how to push indicies onto them? Possibly at the same time as fixing the busted shard? Thanks, Pat ---
**Author:** ![Pat\_Christopher](https://avatars.discourse-cdn.com/v4/letter/p/f14d63/32.png) [@Pat\_Christopher](https://discuss.elastic.co/u/Pat_Christopher)\ **Post date:** [August 10, 2011, 10:33pm UTC](https://discuss.elastic.co/t/non-recovering-index-shard/5117/2 "2011-08-10T22:33:36Z")
I've increased the number of replicas from 1 to 2. This has caused the data to be spread out over all nodes. I did some more research and found that no, I can't change the number of shards for an index after it was created. However, one shard still has a replica which is stuck in initializing. Any idea how I can get ES to abandon that replica and try again someplace else? Or if turn the number of replicas down from 2 to 1 will ES kill the replica that is still initializing and it will be purged? Pat ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 11, 2011, 9:25am UTC](https://discuss.elastic.co/t/non-recovering-index-shard/5117/3 "2011-08-11T09:25:24Z")
> 1. two of the six data nodes have no data on them. I'm not entirely what > they're doing but I'd like to get them involved. Any suggestions on how to > push indicies onto them? Possibly at the same time as fixing the busted > shard? The reason is that rebalancing will not start until the cluster is green in order to reduce the number of relocations. Regarding the stuck shard that is stuck initializing, thats strange... . You have several options here, the simplest would be to bring down the node the replica shard is initializing on, and then start it back up. It should kick the recovery back. Another option is to reduce the replicas to 1, but it won't necessarily remove that initializing shard. Btw, which version are you using? On Thu, Aug 11, 2011 at 1:33 AM, Pat Christopher \< [pat.christopher.hp@gmail.com](mailto:pat.christopher.hp@gmail.com)\> wrote: > I've increased the number of replicas from 1 to 2. This has caused the > data to be spread out over all nodes. I did some more research and found > that no, I can't change the number of shards for an index after it was > created. > > However, one shard still has a replica which is stuck in initializing. Any > idea how I can get ES to abandon that replica and try again someplace else? > Or if turn the number of replicas down from 2 to 1 will ES kill the replica > that is still initializing and it will be purged? > > Pat ---
**Author:** ![Pat\_Christopher](https://avatars.discourse-cdn.com/v4/letter/p/f14d63/32.png) [@Pat\_Christopher](https://discuss.elastic.co/u/Pat_Christopher)\ **Post date:** [August 11, 2011, 5:56pm UTC](https://discuss.elastic.co/t/non-recovering-index-shard/5117/4 "2011-08-11T17:56:03Z")
rebalancing only when green: makes sense. thanks. There was something wrong with that node as a whole, it had two permanently initializing shards after the replica increase. I've shut it down and the cluster has gone green. It had this message over and over in the log file: ``` [Beetle] master should not receive new cluster state from [[O'Meggan, ``` Alfie] The bad node is Beetle and the master for the cluster is O'Meggan, Alfie. Any idea what would cause this? I'm using 0.17.2 Pat ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:57am UTC](https://discuss.elastic.co/t/non-recovering-index-shard/5117/5 "2017-07-06T03:57:24Z")