# ES Cluster Recovery and Restart
**URL:** https://discuss.elastic.co/t/es-cluster-recovery-and-restart/10283
**Category:** Elasticsearch
**Created:** [January 9, 2013, 5:13pm UTC](https://discuss.elastic.co/t/es-cluster-recovery-and-restart/10283 "2013-01-09T17:13:10Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@Gordon\_Tillman](https://discuss.elastic.co/u/Gordon_Tillman)
#### Post date: [January 9, 2013, 5:13pm UTC](https://discuss.elastic.co/t/es-cluster-recovery-and-restart/10283/1 "2013-01-09T17:13:10Z")
Greetings All,
I have a new 6-node ES 0.20.1 cluster. The cluster was up and running with
a small amount of data. There were a total of 302 shards (151 primary,
with 1 copy) distributed amongst 4 indices.
We shut down the cluster so that we could set the ES\_HEAP\_SIZE. After that
we started the cluster. Ever since then it has been in a red state with 30
unassigned shards. We tried tweaking the gateway settings (
[http://www.elasticsearch.org/guide/reference/modules/gateway/](http://www.elasticsearch.org/guide/reference/modules/gateway/)) and
restarting but that didn't help. It has been a full day now with no change
in status.
My thoughts are to do the following:
1. Grep the logs on all of the machines to collect all of the IndexShardMissingException
error messages.
2. Shut down the cluster.
3. See if I can find good copies of the missing shards and, if so, copy
those shard files to the appropriate index directory where the log error
messages reported a problem.
4. Start the cluster and see if it can recover.
Is this a viable plan? Fortunately this is a test cluster so it is not the
end of the world if I have to wipe it and start over. But I want to
understand proper error recovery so I know what to do if this happens in
production.
Also, are there any additional procedural steps I should follow in the
event that I have to restart a cluster so as to avoid this type of issue in
the future?
Many thanks for any information!
Best Regards,
--gordon
--
---
### Author:  [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)
#### Post date: [January 10, 2013, 11:56am UTC](https://discuss.elastic.co/t/es-cluster-recovery-and-restart/10283/2 "2013-01-10T11:56:02Z")
Hello Gordon,
Your plan sounds OK to me, but you have to take care of your recovery
settings. You may have already done this, but I'd put something like:
gateway:
recover\_after\_nodes: 5
recover\_after\_time: 5m
expected\_nodes: 6
To make sure it starts recovery after you've got at least one copy of each
shard available.
Also, if all your nodes are master-eligible, which is the default, you
might want to set discovery.zen.minimum\_master\_nodes to 4 (nodes/2 +1), so
you make sure you don't get split-brain when you restart the whole cluster.
## Best regards, Radu
[http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene
On Wed, Jan 9, 2013 at 7:13 PM, Gordon Tillman [gordyt@gmail.com](mailto:gordyt@gmail.com) wrote:
> Greetings All,
>
> I have a new 6-node ES 0.20.1 cluster. The cluster was up and running
> with a small amount of data. There were a total of 302 shards (151
> primary, with 1 copy) distributed amongst 4 indices.
>
> We shut down the cluster so that we could set the ES\_HEAP\_SIZE. After
> that we started the cluster. Ever since then it has been in a red state
> with 30 unassigned shards. We tried tweaking the gateway settings (
> [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/modules/gateway/)) and
> restarting but that didn't help. It has been a full day now with no change
> in status.
>
> My thoughts are to do the following:
>
> 1. Grep the logs on all of the machines to collect all of the IndexShardMissingException
> error messages.
> 2. Shut down the cluster.
> 3. See if I can find good copies of the missing shards and, if so,
> copy those shard files to the appropriate index directory where the log
> error messages reported a problem.
> 4. Start the cluster and see if it can recover.
>
> Is this a viable plan? Fortunately this is a test cluster so it is not
> the end of the world if I have to wipe it and start over. But I want to
> understand proper error recovery so I know what to do if this happens in
> production.
>
> Also, are there any additional procedural steps I should follow in the
> event that I have to restart a cluster so as to avoid this type of issue in
> the future?
>
> Many thanks for any information!
>
> Best Regards,
>
> --gordon
>
> --
--
---
### Author:  [@Gordon\_Tillman](https://discuss.elastic.co/u/Gordon_Tillman)
#### Post date: [January 10, 2013, 2:10pm UTC](https://discuss.elastic.co/t/es-cluster-recovery-and-restart/10283/3 "2013-01-10T14:10:34Z")
Radu thank you very much for the information. I appreciate your time and
trouble.
--gordon
On Thursday, January 10, 2013 5:56:02 AM UTC-6, Radu Gheorghe wrote:
> Hello Gordon,
>
> Your plan sounds OK to me, but you have to take care of your recovery
> settings. You may have already done this, but I'd put something like:
>
> gateway:
>
> ```
> recover_after_nodes: 5
> recover_after_time: 5m
>
> expected_nodes: 6
>
> ```
>
> To make sure it starts recovery after you've got at least one copy of each
> shard available.
>
> Also, if all your nodes are master-eligible, which is the default, you
> might want to set discovery.zen.minimum\_master\_nodes to 4 (nodes/2 +1),
> so you make sure you don't get split-brain when you restart the whole
> cluster.
>
> ## Best regards, Radu
>
> [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene
>
> On Wed, Jan 9, 2013 at 7:13 PM, Gordon Tillman \<[gor...@gmail.com](mailto:gor...@gmail.com)\
>
> > wrote:
>
> > Greetings All,
> >
> > I have a new 6-node ES 0.20.1 cluster. The cluster was up and running
> > with a small amount of data. There were a total of 302 shards (151
> > primary, with 1 copy) distributed amongst 4 indices.
> >
> > We shut down the cluster so that we could set the ES\_HEAP\_SIZE. After
> > that we started the cluster. Ever since then it has been in a red state
> > with 30 unassigned shards. We tried tweaking the gateway settings (
> > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/gateway/)) and
> > restarting but that didn't help. It has been a full day now with no change
> > in status.
> >
> > My thoughts are to do the following:
> >
> > 1. Grep the logs on all of the machines to collect all of the IndexShardMissingException
> > error messages.
> > 2. Shut down the cluster.
> > 3. See if I can find good copies of the missing shards and, if so,
> > copy those shard files to the appropriate index directory where the log
> > error messages reported a problem.
> > 4. Start the cluster and see if it can recover.
> >
> > Is this a viable plan? Fortunately this is a test cluster so it is not
> > the end of the world if I have to wipe it and start over. But I want to
> > understand proper error recovery so I know what to do if this happens in
> > production.
> >
> > Also, are there any additional procedural steps I should follow in the
> > event that I have to restart a cluster so as to avoid this type of issue in
> > the future?
> >
> > Many thanks for any information!
> >
> > Best Regards,
> >
> > --gordon
> >
> > --
--
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 2:57am UTC](https://discuss.elastic.co/t/es-cluster-recovery-and-restart/10283/4 "2017-07-06T02:57:00Z")