# Cluster won't recover

**URL:** <https://discuss.elastic.co/t/cluster-wont-recover/81684>\
**Category:** Elasticsearch\
**Created:** [April 8, 2017, 5:25pm UTC](https://discuss.elastic.co/t/cluster-wont-recover/81684 "2017-04-08T17:25:49Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Or\_Arnon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/or_arnon/32/43677_2.png) [@Or\_Arnon](https://discuss.elastic.co/u/Or_Arnon)\
**Post date:** [April 8, 2017, 5:25pm UTC](https://discuss.elastic.co/t/cluster-wont-recover/81684/1 "2017-04-08T17:25:50Z")

</div>

IllegalIndexShardStateException[CurrentState[POST\_RECOVERY] operation only allowed when started/recovering, origin [PRIMARY]]

We have 8 nodes, 4 are spots and 4 are on demand. The cluster keeps crashing when with the above error when we start our on demand instances which contain the primary shards of the cluster

We're using Elasticsearch 2.4  
Nodes are r3.2xlarge, r4.2xlarge, r3.4xlarge  
Heap is configured to 31GB (half of the node's RAM)

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [April 8, 2017, 11:01pm UTC](https://discuss.elastic.co/t/cluster-wont-recover/81684/2 "2017-04-08T23:01:52Z")

</div>

Can you post the entire error?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 9, 2017, 8:46am UTC](https://discuss.elastic.co/t/cluster-wont-recover/81684/3 "2017-04-09T08:46:44Z")

</div>

Having nodes come and go in a cluster will cause data to be redistributed. Using spot instances for data nodes can therefore be difficult. What is the reason behind using spot instances? How much data do you have and how many replicas does each index have? Are you using [shard allocation filtering](https://www.elastic.co/guide/en/elasticsearch/reference/5.3/allocation-filtering.html) and or [shard allocation awareness](https://www.elastic.co/guide/en/elasticsearch/reference/5.3/allocation-awareness.html) to control the distribution of primary and replica shards across the cluster?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 7, 2017, 8:51am UTC](https://discuss.elastic.co/t/cluster-wont-recover/81684/4 "2017-05-07T08:51:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.