# Gateway.recover\_after\_nodes question **URL:** \ **Category:** Elasticsearch\ **Created:** [September 15, 2010, 4:38pm UTC](https://discuss.elastic.co/t/gateway-recover-after-nodes-question/3339 "2010-09-15T16:38:23Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Andrei](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrei/32/2856_2.png) [@Andrei](https://discuss.elastic.co/u/Andrei)\ **Post date:** [September 15, 2010, 4:38pm UTC](https://discuss.elastic.co/t/gateway-recover-after-nodes-question/3339/1 "2010-09-15T16:38:23Z")
I'm trying to understand how to use gateway.recover\_after\_nodes setting. I have a 2 node cluster with the following config: cluster: name: Awesome gateway: type: s3 s3: bucket: \*\*\*\* recover\_after\_nodes: 2 index: gateway: snapshot\_interval: 60s cloud: aws: access\_key: \*\*\*\* secret\_key: \*\*\*\* discovery: type: ec2 I created an index with 2 shards and 1 replica and indexed a few documents into it. Then I shut down the cluster and brought up both nodes, expecting them to recover from local index storage, but the log says: [16:34:39,657][INFO][cluster.service] [Black King] added {[Goldeneye][e8881b4f-7458-4a66-b590-c612da5cc54e][inet[/ 10.204.250.116:9300]],}, reason: zen-disco-receive(from node[[Goldeneye][e8881b4f-7458-4a66-b590-c612da5cc54e][inet[/ 10.204.250.116:9300]]]) [16:34:40,988][INFO][cluster.metadata] [Black King] [foo] creating index, cause [gateway], shards [2]/[1], mappings [bar] [16:34:42,879][INFO][cluster.metadata] [Black King] [foo] created and added to cluster\_state My understanding was that recover\_after\_nodes = 2 gives the cluster a chance to recover index from local storage, but apparently not? -Andrei ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 15, 2010, 5:47pm UTC](https://discuss.elastic.co/t/gateway-recover-after-nodes-question/3339/2 "2010-09-15T17:47:41Z")
recover\_after\_nodes only means that the actual state recovery (which indices were created) as well as shard allocation (when using shared gateway) will happen only after the specific number of nodes has been started in a cluster. This makes sense in shared storage gateway mode so locally stored shards can be reused (shards gets allocated to nodes that have the smallest diff between their local storage and the shared gateway). -shay.banon On Wed, Sep 15, 2010 at 6:38 PM, Andrei [andrei@zmievski.org](mailto:andrei@zmievski.org) wrote: > I'm trying to understand how to use gateway.recover\_after\_nodes > setting. I have a 2 node cluster with the following config: > > cluster: > name: Awesome > > gateway: > type: s3 > s3: > bucket: \*\*\*\* > recover\_after\_nodes: 2 > > index: > gateway: > snapshot\_interval: 60s > > cloud: > aws: > access\_key: \*\*\*\* > secret\_key: \*\*\*\* > > discovery: > type: ec2 > > I created an index with 2 shards and 1 replica and indexed a few > documents into it. Then I shut down the cluster and brought up both > nodes, expecting them to recover from local index storage, but the log > says: > > [16:34:39,657][INFO][cluster.service] [Black King] added > {[Goldeneye][e8881b4f-7458-4a66-b590-c612da5cc54e][inet[/ > 10.204.250.116:9300]],}, reason: zen-disco-receive(from > node[[Goldeneye][e8881b4f-7458-4a66-b590-c612da5cc54e][inet[/ > 10.204.250.116:9300]]]) > [16:34:40,988][INFO][cluster.metadata] [Black King] [foo] > creating index, cause [gateway], shards [2]/[1], mappings [bar] > [16:34:42,879][INFO][cluster.metadata] [Black King] [foo] > created and added to cluster\_state > > My understanding was that recover\_after\_nodes = 2 gives the cluster a > chance to recover index from local storage, but apparently not? > > -Andrei ---
**Author:** ![Andrei](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrei/32/2856_2.png) [@Andrei](https://discuss.elastic.co/u/Andrei)\ **Post date:** [September 15, 2010, 10:09pm UTC](https://discuss.elastic.co/t/gateway-recover-after-nodes-question/3339/3 "2010-09-15T22:09:15Z")
Sorry, I am still trying to wrap my head around this. In my case, the s3 gateway is a shared one (correct?), so I was trying to get the nodes to recover the state/shards from the local shards, and only then fetch the shards from S3 if the local recovery was not possible. Did I miss something? What is the default value of recovery\_after\_nodes? In my 2 node cluster, what is the advantage of setting recovery\_after\_nodes to 2 instead of 1? Are there guidelines for what it should be, depending on the number of nodes and shards? Overall I'm really impressed by the dynamic node discovery and failover capabilities of ES. Thanks for the great work. -Andrei On Sep 15, 10:47 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > recover\_after\_nodes only means that the actual state recovery (which indices > were created) as well as shard allocation (when using shared gateway) will > happen only after the specific number of nodes has been started in a > cluster. > > This makes sense in shared storage gateway mode so locally stored shards can > be reused (shards gets allocated to nodes that have the smallest diff > between their local storage and the shared gateway). > > -shay.banon ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 15, 2010, 10:22pm UTC](https://discuss.elastic.co/t/gateway-recover-after-nodes-question/3339/4 "2010-09-15T22:22:04Z")
Yea, s3 gateway is a shared one. The recovery process after a full shutdown always goes to the gateway. The way that the "reuse local storage" mode works is that, after "recover\_after\_nodes" have been discovered, when trying to allocate a specific shard to a node, a diff is one between what data the gateway has between all the nodes that hold that shard data, and then the shard is allocated to the node with the "closest" amount of data. This means that less data will need to be fetched from the gateway in order to complete the recovery process as the local data on the node can be reused. In general, the closer the recover\_after\_nodes is to the actual amount of nodes you are going to start, the better chances that the recovery process will complete quickly, as there will be more nodes to check for possible shard allocate and reuse of the local storage. The local gateway feature in master works completely different. Recovery always happen only from the local storage, there is no remote storage in question. It is much faster and more resilient to corruptions, at the cost of relying on local storage and replication to provide durability (and not a shared storage). Once 0.11 is out I plan to write a whole new section in the docs, which is more of "vertical" documentation and not module based documentation as the format of the docs is today. In this section, I plan to cover things like Gateway (long term persistency), Analysis Paralysis, and other "vertical" aspects. -shay.banon On Thu, Sep 16, 2010 at 12:09 AM, Andrei [andrei@zmievski.org](mailto:andrei@zmievski.org) wrote: > Sorry, I am still trying to wrap my head around this. > > In my case, the s3 gateway is a shared one (correct?), so I was trying > to get the nodes to recover the state/shards from the local shards, > and only then fetch the shards from S3 if the local recovery was not > possible. Did I miss something? > > What is the default value of recovery\_after\_nodes? In my 2 node > cluster, what is the advantage of setting recovery\_after\_nodes to 2 > instead of 1? Are there guidelines for what it should be, depending on > the number of nodes and shards? > > Overall I'm really impressed by the dynamic node discovery and > failover capabilities of ES. Thanks for the great work. > > -Andrei > > On Sep 15, 10:47 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > recover\_after\_nodes only means that the actual state recovery (which > > indices > > were created) as well as shard allocation (when using shared gateway) > > will > > happen only after the specific number of nodes has been started in a > > cluster. > > > > This makes sense in shared storage gateway mode so locally stored shards > > can > > be reused (shards gets allocated to nodes that have the smallest diff > > between their local storage and the shared gateway). > > > > -shay.banon ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:19am UTC](https://discuss.elastic.co/t/gateway-recover-after-nodes-question/3339/5 "2017-07-06T04:19:17Z")