# How does a recovering node validate any shard information/data during recover? **URL:** \ **Category:** Elasticsearch\ **Created:** [August 1, 2011, 5:34am UTC](https://discuss.elastic.co/t/how-does-a-recovering-node-validate-any-shard-information-data-during-recover/5013 "2011-08-01T05:34:41Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Paul\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_smith/32/1323_2.png) [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)\ **Post date:** [August 1, 2011, 5:34am UTC](https://discuss.elastic.co/t/how-does-a-recovering-node-validate-any-shard-information-data-during-recover/5013/1 "2011-08-01T05:34:41Z")
Had an interesting question I couldn't answer today. When an existing node of a cluster is shutdown, or it is terminated, when it starts up again, it will goes through a recovery phase. Just wanted to understand a little bit more about what checks a node does to check whether it has valid state, and how it rebuilds it's shard meta-data and content if it's not. In a fairly static index that doesn't change, is there a fast path to confirm that said recovering node has indeed got the latest info and there's no need to re-replicate anything? Are existing replicas used as a source for copying info, or does it always favour retrieving from the master shard node (for a given shard)? This is the meat and potatoes of what ES is doing for us, and on the one hand, "we shouldn't need to know", but... thanks, Paul ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 1, 2011, 6:38am UTC](https://discuss.elastic.co/t/how-does-a-recovering-node-validate-any-shard-information-data-during-recover/5013/2 "2011-08-01T06:38:41Z")
If a single node fails, then a few things happen. First, if on that node there were some elected primary shards, then a replica shard on the rest of the nodes will be promoted to primary. Also, the shards allocated on that node will begin to be allocated on the rest of the nodes (throttled). When the node comes back, then it will join the cluster, and, will start to be assigned shards. If not all shards from that node have been assigned yet to the rest of the cluster, then they will favor to be allocated on that node, if they have similar data (checksumed index file). In general, the above logic is more generic in elasticsearch. When a replica shard is going through the allocation logic, it will favor first to be allocated on a node that has the closest matching index files against the relevant primary shard, so to reduce the amount of data to transfer from the primary shard to the replica shard. On Mon, Aug 1, 2011 at 8:34 AM, Paul Smith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) wrote: > Had an interesting question I couldn't answer today. When an existing node > of a cluster is shutdown, or it is terminated, when it starts up again, it > will goes through a recovery phase. Just wanted to understand a little bit > more about what checks a node does to check whether it has valid state, and > how it rebuilds it's shard meta-data and content if it's not. > > In a fairly static index that doesn't change, is there a fast path to > confirm that said recovering node has indeed got the latest info and there's > no need to re-replicate anything? Are existing replicas used as a source > for copying info, or does it always favour retrieving from the master shard > node (for a given shard)? > > This is the meat and potatoes of what ES is doing for us, and on the one > hand, "we shouldn't need to know", but... > > thanks, > > Paul ---
**Author:** ![Paul\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_smith/32/1323_2.png) [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)\ **Post date:** [August 1, 2011, 7:02am UTC](https://discuss.elastic.co/t/how-does-a-recovering-node-validate-any-shard-information-data-during-recover/5013/3 "2011-08-01T07:02:00Z")
> When the node comes back, then it will join the cluster, and, will start > to be assigned shards. If not all shards from that node have been assigned > yet to the rest of the cluster, then they will favor to be allocated on that > node, if they have similar data (checksumed index file). If a node goes down for an extended period, and presuming that the shard info is now pretty much out of date, I'm presuming there's some point it goes "ok stuff it, 'rm -rf' and get a fresh copy from the primary? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 1, 2011, 7:10am UTC](https://discuss.elastic.co/t/how-does-a-recovering-node-validate-any-shard-information-data-during-recover/5013/4 "2011-08-01T07:10:26Z")
On Mon, Aug 1, 2011 at 10:02 AM, Paul Smith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) wrote: > > When the node comes back, then it will join the cluster, and, will start > > to be assigned shards. If not all shards from that node have been assigned > > yet to the rest of the cluster, then they will favor to be allocated on that > > node, if they have similar data (checksumed index file). > > If a node goes down for an extended period, and presuming that the shard > info is now pretty much out of date, I'm presuming there's some point it > goes "ok stuff it, 'rm -rf' and get a fresh copy from the primary? Yes, effectively this will happen as the data in the "live" nodes will start to diverge from what was stored in the shards that exist on the node that went down. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:58am UTC](https://discuss.elastic.co/t/how-does-a-recovering-node-validate-any-shard-information-data-during-recover/5013/5 "2017-07-06T03:58:42Z")