# Shards stuck in INITIALIZING state after CLUSTER restart

**URL:** <https://discuss.elastic.co/t/shards-stuck-in-initializing-state-after-cluster-restart/113394>\
**Category:** Elasticsearch\
**Created:** [December 28, 2017, 5:57am UTC](https://discuss.elastic.co/t/shards-stuck-in-initializing-state-after-cluster-restart/113394 "2017-12-28T05:57:55Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vijaysimha\_Naik](https://avatars.discourse-cdn.com/v4/letter/v/a3d4f5/32.png) [@Vijaysimha\_Naik](https://discuss.elastic.co/u/Vijaysimha_Naik)\
**Post date:** [December 28, 2017, 5:57am UTC](https://discuss.elastic.co/t/shards-stuck-in-initializing-state-after-cluster-restart/113394/1 "2017-12-28T05:57:55Z")

</div>

Elasticsearch version (bin/elasticsearch --version): 1.1.1

Plugins installed: []

JVM version (java -version):openjdk version "1.8.0\_151"

OS version (uname -a if on a Unix-like system): x86\_64 x86\_64 x86\_64 GNU/Linux

I have a 2 NODE, 3 INDEX elastic cluster. Each Index has 5 Primary shards with 1 replica for each Primary. I restarted the cluster yesterday since some of the shards had gone UNASSIGNED due to a disk space issue. Now the disk issue has been resolved and the cluster is working good after restart. However, there are 4 shards in INITIALIZING state since the past 12 hours or so. I am frequently seeing the following in logs:

```auto
[2017-12-28 05:02:13,797][WARN][cluster.action.shard] [ES_PROD] [intellinote][2] received shard failed for [intellinote][2], node[XuI6fDcvQfuclZNydldeIA], [P], s[INITIALIZING], indexUUID [sO34VtN-S6KrnIpu4s-c5g], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[intellinote][2] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[intellinote][2] shard allocated for local recovery (post api), should exist, but doesn't, current files: 

```

These logs are frequently generated for all the 4 shards which are trying to INITIALIZE. Is there a way to make them STARTED without any loss of data? Any help would be greatly appreciated.

Thanks,  
Vijay

---

<div class="post-metadata">

**Author:** ![Fram\_Souza](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fram_souza/32/24833_2.png) [@Fram\_Souza](https://discuss.elastic.co/u/Fram_Souza)\
**Post date:** [December 29, 2017, 6:03pm UTC](https://discuss.elastic.co/t/shards-stuck-in-initializing-state-after-cluster-restart/113394/2 "2017-12-29T18:03:48Z")

</div>

Hi @Vijaysimha_Naik

Are all shards components of the same index? Change the number of replicas to 0, wait for the cluster to become available, and change to 1 again.

I sent here the output of the command:  
curl localhost:9200/\_cat /shards 2\> /dev/null | grep INI

---

<div class="post-metadata">

**Author:** ![Vijaysimha\_Naik](https://avatars.discourse-cdn.com/v4/letter/v/a3d4f5/32.png) [@Vijaysimha\_Naik](https://discuss.elastic.co/u/Vijaysimha_Naik)\
**Post date:** [January 5, 2018, 10:13pm UTC](https://discuss.elastic.co/t/shards-stuck-in-initializing-state-after-cluster-restart/113394/3 "2018-01-05T22:13:34Z")

</div>

Hi @Fram_Souza.. Thanks for the suggestion. I solved the issue by deleting the segments.gen file from the /index directory and restarted the cluster. Also, I did a reroute command on the replica shards.

Thanks for your help.

Regards,  
Vijay

---

<div class="post-metadata">

**Author:** ![Fram\_Souza](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fram_souza/32/24833_2.png) [@Fram\_Souza](https://discuss.elastic.co/u/Fram_Souza)\
**Post date:** [January 10, 2018, 4:39pm UTC](https://discuss.elastic.co/t/shards-stuck-in-initializing-state-after-cluster-restart/113394/4 "2018-01-10T16:39:23Z")

</div>

Delete the segments.gen is a bad ideia and NEVER should to be used.

For solving this problem, you can change number of replicas in cluster.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 7, 2018, 4:39pm UTC](https://discuss.elastic.co/t/shards-stuck-in-initializing-state-after-cluster-restart/113394/5 "2018-02-07T16:39:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.