# total\_shards\_per\_node and node failure **URL:** \ **Category:** Elasticsearch\ **Created:** [December 5, 2012, 6:39pm UTC](https://discuss.elastic.co/t/total-shards-per-node-and-node-failure/9957 "2012-12-05T18:39:50Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Jeff\_Rick](https://avatars.discourse-cdn.com/v4/letter/j/e9c0ed/32.png) [@Jeff\_Rick](https://discuss.elastic.co/u/Jeff_Rick)\ **Post date:** [December 5, 2012, 6:39pm UTC](https://discuss.elastic.co/t/total-shards-per-node-and-node-failure/9957/1 "2012-12-05T18:39:50Z")
Hi, Had a question on the setting "total\_shards\_per\_node". If I set an index up to have: -- shards = # of nodes in my cluster -- total\_shards\_per\_node of 2 -- replicas set to 1 Then I would get an even distribution of shards and replicas across my cluster, make sense. The question is what happens if I have a node failure? Does the cluster fail to allocate new replicas for the shards that were on the failed node? Does that mean I have potential data loss if any two nodes fail, no matter the size of the cluster? I did some digging in the code and that seems to be the case but wanted to confirm I haven't missed anything. Thanks -- ---
**Author:** ![Dan\_Lecocq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dan_lecocq/32/2565_2.png) [@Dan\_Lecocq](https://discuss.elastic.co/u/Dan_Lecocq)\ **Post date:** [December 5, 2012, 6:55pm UTC](https://discuss.elastic.co/t/total-shards-per-node-and-node-failure/9957/2 "2012-12-05T18:55:54Z")
My understanding is that while you wouldn't get data /loss/, the shards on that node would be unavailable while that node is unavailable. Since those shards exist on other nodes as well, they can still be queries and data inserted into them. When a replacement node is added, they can be used to restore the unassigned shards on the new node. On Wednesday, December 5, 2012 10:39:50 AM UTC-8, Jeff Rick wrote: > Hi, > > Had a question on the setting "total\_shards\_per\_node". If I set an index > up to have: > > -- shards = # of nodes in my cluster > -- total\_shards\_per\_node of 2 > -- replicas set to 1 > > Then I would get an even distribution of shards and replicas across my > cluster, make sense. The question is what happens if I have a node > failure? Does the cluster fail to allocate new replicas for the shards > that were on the failed node? Does that mean I have potential data loss if > any two nodes fail, no matter the size of the cluster? > > I did some digging in the code and that seems to be the case but wanted to > confirm I haven't missed anything. > > Thanks -- ---
**Author:** ![Jeff\_Rick](https://avatars.discourse-cdn.com/v4/letter/j/e9c0ed/32.png) [@Jeff\_Rick](https://discuss.elastic.co/u/Jeff_Rick)\ **Post date:** [December 5, 2012, 8:07pm UTC](https://discuss.elastic.co/t/total-shards-per-node-and-node-failure/9957/3 "2012-12-05T20:07:22Z")
So understand that the data would still be there when the node came back, but if you lost one node that had the primary shard, it won't be able to create a new replica, correct? If that is the case then if you lost two nodes 1 that had the primary and then later lost a second node that had the replica you would experience data loss. Thanks On Wednesday, December 5, 2012 1:55:54 PM UTC-5, Dan Lecocq wrote: > My understanding is that while you wouldn't get data /loss/, the shards on > that node would be unavailable while that node is unavailable. Since those > shards exist on other nodes as well, they can still be queries and data > inserted into them. When a replacement node is added, they can be used to > restore the unassigned shards on the new node. > > On Wednesday, December 5, 2012 10:39:50 AM UTC-8, Jeff Rick wrote: > > > Hi, > > > > Had a question on the setting "total\_shards\_per\_node". If I set an index > > up to have: > > > > -- shards = # of nodes in my cluster > > -- total\_shards\_per\_node of 2 > > -- replicas set to 1 > > > > Then I would get an even distribution of shards and replicas across my > > cluster, make sense. The question is what happens if I have a node > > failure? Does the cluster fail to allocate new replicas for the shards > > that were on the failed node? Does that mean I have potential data loss if > > any two nodes fail, no matter the size of the cluster? > > > > I did some digging in the code and that seems to be the case but wanted > > to confirm I haven't missed anything. > > > > Thanks -- ---
**Author:** ![Dan\_Lecocq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dan_lecocq/32/2565_2.png) [@Dan\_Lecocq](https://discuss.elastic.co/u/Dan_Lecocq)\ **Post date:** [December 5, 2012, 8:14pm UTC](https://discuss.elastic.co/t/total-shards-per-node-and-node-failure/9957/4 "2012-12-05T20:14:46Z")
If you lose the primary shard, the replica can be promoted to be the primary. I'm not entirely sure of when ES makes that decision, but it does happen. So you could bring in a completely blank node as a replacement, and it would generate a replica of the lost shard from the newly-promoted primary. Yes, if you lost the primary and lost the replica before you bring in a replacement, then you'd lose the data. That is, supposing that you don't have a gateway set up to restore it. What kind of node loss in particular are you thinking about? That the node completely disappears? Or completely dies? Or just if it stops elasticsearch / etc.? On Wednesday, December 5, 2012 12:07:22 PM UTC-8, Jeff Rick wrote: > So understand that the data would still be there when the node came back, > but if you lost one node that had the primary shard, it won't be able to > create a new replica, correct? If that is the case then if you lost two > nodes 1 that had the primary and then later lost a second node that had the > replica you would experience data loss. > > Thanks > > On Wednesday, December 5, 2012 1:55:54 PM UTC-5, Dan Lecocq wrote: > > > My understanding is that while you wouldn't get data /loss/, the shards > > on that node would be unavailable while that node is unavailable. Since > > those shards exist on other nodes as well, they can still be queries and > > data inserted into them. When a replacement node is added, they can be used > > to restore the unassigned shards on the new node. > > > > On Wednesday, December 5, 2012 10:39:50 AM UTC-8, Jeff Rick wrote: > > > > > Hi, > > > > > > Had a question on the setting "total\_shards\_per\_node". If I set an > > > index up to have: > > > > > > -- shards = # of nodes in my cluster > > > -- total\_shards\_per\_node of 2 > > > -- replicas set to 1 > > > > > > Then I would get an even distribution of shards and replicas across my > > > cluster, make sense. The question is what happens if I have a node > > > failure? Does the cluster fail to allocate new replicas for the shards > > > that were on the failed node? Does that mean I have potential data loss if > > > any two nodes fail, no matter the size of the cluster? > > > > > > I did some digging in the code and that seems to be the case but wanted > > > to confirm I haven't missed anything. > > > > > > Thanks -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:01am UTC](https://discuss.elastic.co/t/total-shards-per-node-and-node-failure/9957/5 "2017-07-06T03:01:09Z")