# Serious Availability Challenges w/ Replica Recovery Process in Cloud Environments **URL:** \ **Category:** Elasticsearch\ **Created:** [August 26, 2014, 12:38pm UTC](https://discuss.elastic.co/t/serious-availability-challenges-w-replica-recovery-process-in-cloud-environments/19462 "2014-08-26T12:38:14Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![Nariman\_Haghighi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nariman_haghighi/32/1065_2.png) [@Nariman\_Haghighi](https://discuss.elastic.co/u/Nariman_Haghighi)\ **Post date:** [August 26, 2014, 12:38pm UTC](https://discuss.elastic.co/t/serious-availability-challenges-w-replica-recovery-process-in-cloud-environments/19462/1 "2014-08-26T12:38:14Z")
I've read as much as I can get my hands on with respect to cluster restarts and optimizing replica recovery [1] on 1.3.2 and I still think there's a big problem here. Scenario: - 2 node cluster running on Azure (2 large nodes: 7GB RAM, 4 cores, 400 Mbps network) - 600GB disk on each side, current cluster is using about 25GB of that on each node - Cluster must be resilient to index requests at all times! This is a core requirement that we can't get around. What works pretty reliably in this scenario is that after each node is restarted, you wait for the full sync before upgrading the other node (GREEN status). Anything else will cause data loss. The trouble with this approach is we are dealing with network capacity of 400 Mbps and while 25 GB can be transferred in a reasonable time... 600GB definitely can't. On Azure, you have a max of _15 minutes_ OnStart to make sure the instance is healthy again before the fabric controller recycles it for you. And I think that limit is reasonable. Sure we could go up 1 size to Extra Large nodes that offer a 800 Mbps network but still this isn't going to get us to the point where we can maintain 100% availability of a 600GB cluster on 2 nodes. There is a bit of light at the end of the tunnel: We plan on assigning a sequence number to operations that occur on primary > shards.This is a really interesting feature that will lay the groundwork > for many future features in Elasticsearch. The most obvious one is speeding > up the replica recovery process when a node is restarted. Currently we have > to copy every segment which is different which, over time, means every > segment! Sequence numbers will allow us to copy over only the data that has > really changed. It couldn't come soon enough for us. The current replica recovery process is really untenable in cloud environments that carry heavy INDEX workloads. Any thoughts on when this can be expected or if there are viable workarounds in the meantime would be greatly appreciated. [1] - Research: > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. [http://stackoverflow.com/questions/24696020/elasticsearch-node-recovery-cluster-restart-correct-settings](http://stackoverflow.com/questions/24696020/elasticsearch-node-recovery-cluster-restart-correct-settings) [https://groups.google.com/forum/#!searchin/elasticsearch/cluster$20restart$20disable$20allocation/elasticsearch/csziKQPBauU/9PKWbkhJ50IJ](https://groups.google.com/forum/#!searchin/elasticsearch/cluster$20restart$20disable$20allocation/elasticsearch/csziKQPBauU/9PKWbkhJ50IJ) [http://elasticsearch-users.115913.n3.nabble.com/Restarting-an-active-node-without-needing-to-recover-all-data-remotely-td4039346.html](http://elasticsearch-users.115913.n3.nabble.com/Restarting-an-active-node-without-needing-to-recover-all-data-remotely-td4039346.html) > [https://groups.google.com/forum/#!searchin/elasticsearch/cluster$20restart/elasticsearch/lN6copl0Bzk/RK5ESX8nu-8J](https://groups.google.com/forum/#!searchin/elasticsearch/cluster$20restart/elasticsearch/lN6copl0Bzk/RK5ESX8nu-8J) [https://groups.google.com/forum/#!searchin/elasticsearch/cluster$20restart/elasticsearch/plTGgtE\_YCU/fCyd2elxkAsJ](https://groups.google.com/forum/#!searchin/elasticsearch/cluster$20restart/elasticsearch/plTGgtE_YCU/fCyd2elxkAsJ) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0a9631f6-6a59-413c-ba0e-2c161ebea10e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0a9631f6-6a59-413c-ba0e-2c161ebea10e%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:06am UTC](https://discuss.elastic.co/t/serious-availability-challenges-w-replica-recovery-process-in-cloud-environments/19462/2 "2017-07-06T01:06:28Z")
Closed on July 6, 2017, 1:06am UTC