# Recovery

**URL:** <https://discuss.elastic.co/t/recovery/3090>\
**Category:** Elasticsearch\
**Created:** [July 12, 2010, 7:10pm UTC](https://discuss.elastic.co/t/recovery/3090 "2010-07-12T19:10:38Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![James\_Cook](https://avatars.discourse-cdn.com/v4/letter/j/898d66/32.png) [@James\_Cook](https://discuss.elastic.co/u/James_Cook)\
**Post date:** [July 12, 2010, 7:10pm UTC](https://discuss.elastic.co/t/recovery/3090/1 "2010-07-12T19:10:38Z")

</div>

Assume the complete failure of all my instances of ES on EC2 using the  
new cloud plugin for discovery and gateway.

My management console will attempt to bring up instances to replace  
the ones which failed.

Can you describe the restoration process that begins as that first  
node is restarted and subsequent nodes come back online?

Is there any manual intervention required by the process?

If only the data which remains is in S3 (each instance of ES failed  
and took its memory and hard drive storage with it), does this hurt  
recovery?

Will the system appear offline until recovery is completed?

Thanks for making search fun!

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [July 12, 2010, 7:41pm UTC](https://discuss.elastic.co/t/recovery/3090/2 "2010-07-12T19:41:54Z")

</div>

If all the instances fail, and you start new ondes, then data will be  
recovered from s3. While data is being recovered (note, this might take  
time, depends on s3 and the machine IO performance), the cluster will not be  
available. Once a specific shard has recovered, then it will be available,  
you will get partial search results (the search returns how many shards  
succeeded and how many failed).

Note, in 0.9, hopefully, the recovery from s3 will be a bit faster (working  
on a native implementation of s3 which should be faster than and less  
resource hog than all the current ones out there).

A quick note regarding 0.9 and the upcoming reuse work dir feature. If the  
machines fail, but file system is still there (assuming you store the index  
on the file system), then data that already exists on the local file system  
will not be recovered from s3, resulting in faster recovery. For this  
feature, you would probably want to set to `gateway.recover_after_nodes`  
setting, so the recovery process will only start after N nodes have started  
(only relevant after full cluster shutdown / failure).

-shay.banon

On Mon, Jul 12, 2010 at 10:10 PM, oravecz [jcook@tracermedia.com](mailto:jcook@tracermedia.com) wrote:

> Assume the complete failure of all my instances of ES on EC2 using the  
> new cloud plugin for discovery and gateway.
> 
> My management console will attempt to bring up instances to replace  
> the ones which failed.
> 
> Can you describe the restoration process that begins as that first  
> node is restarted and subsequent nodes come back online?
> 
> Is there any manual intervention required by the process?
> 
> If only the data which remains is in S3 (each instance of ES failed  
> and took its memory and hard drive storage with it), does this hurt  
> recovery?
> 
> Will the system appear offline until recovery is completed?
> 
> Thanks for making search fun!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:22am UTC](https://discuss.elastic.co/t/recovery/3090/3 "2017-07-06T04:22:25Z")

</div>

Closed on July 6, 2017, 4:22am UTC