# Multi node fail **URL:** \ **Category:** Elasticsearch\ **Created:** [August 8, 2020, 9:29am UTC](https://discuss.elastic.co/t/multi-node-fail/244244 "2020-08-08T09:29:50Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![cadirol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cadirol/32/38459_2.png) [@cadirol](https://discuss.elastic.co/u/cadirol)\ **Post date:** [August 8, 2020, 9:29am UTC](https://discuss.elastic.co/t/multi-node-fail/244244/1 "2020-08-08T09:29:50Z")
Hi All For operational continuity in an multinode cluster i have two questions in my head: the cluster contains, 14 nodes, 7 nodes on site A, 7 nodes on site B, ES Version 6.4 What if a) one site goes down unexpected and can be restored after lets say power fail? In this case the cluster routing is set to "all" all the time. Does elasticsearch recover by its self? and b) one site goes down in a planned work and the routing is set to "none" in advance? Of course, all the shards have at least one replica and are well distributed over the cluster. some days ago, we had situation b) because of a relocation of site B to site C. After relocation and booting the nodes on site C, all the nodes which should reallocating unassigned shards, was throttled according to the recovery settings (simultaneous recovery was set to 8). the only thing we could do, to go in a green state, was reduce the affected shard replicas to 0. What is the correct way, to relocate 50% of all the nodes? Buy new nodes, place these on site C, add nodes on site C to the cluster, and shutdown nodes on site B? shutdown all nodes on site B without disabling routing? Thank you very much for your inputs! A ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 8, 2020, 9:40am UTC](https://discuss.elastic.co/t/multi-node-fail/244244/2 "2020-08-08T09:40:40Z")
In a correctly configured cluster you need 3 zones to achieve high availability as a strict majority of master eligible nodes need to be available. With 14 nodes split evenly across two zones your cluster should go into read only mode if one full site is list unless you have more master eligible nodes in the x Zac one that stayed up. Make sure minimum\_master\_nodes is set correctly as you otherwise could experience data loss without even realising it. What f all your nodes are master eligible it should be set to 8 as that is the majority of 14. I would recommend adding a dedicated master node as tie breaker in a third zone. ---
**Author:** ![cadirol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cadirol/32/38459_2.png) [@cadirol](https://discuss.elastic.co/u/cadirol)\ **Post date:** [August 8, 2020, 9:58am UTC](https://discuss.elastic.co/t/multi-node-fail/244244/3 "2020-08-08T09:58:30Z")
Hi @Christian_Dahlqvist You are absolute right about the master nodes! and of course, we have 3 Master nodes on a complete different Infrastructure running. For my two scenarios a) and b) above we can say, all the 3 Master nodes stay reachable. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 8, 2020, 10:04am UTC](https://discuss.elastic.co/t/multi-node-fail/244244/4 "2020-08-08T10:04:06Z")
Are you using shard as location awareness to make sure shards are distributed across the zones? ---
**Author:** ![cadirol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cadirol/32/38459_2.png) [@cadirol](https://discuss.elastic.co/u/cadirol)\ **Post date:** [August 8, 2020, 10:06am UTC](https://discuss.elastic.co/t/multi-node-fail/244244/5 "2020-08-08T10:06:36Z")
Yes I've set `node.attr.rack_id: site_a` and `node.attr.rack_id: site_b` according to the site, on which the node is running ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 8, 2020, 10:11am UTC](https://discuss.elastic.co/t/multi-node-fail/244244/6 "2020-08-08T10:11:23Z")
If you have allocation awareness set up the replica shards will not get reassigned whole zone B is down so indices will be yellow. Once the zone comes back up the shards will need to be brought up to speed. Hi we’d long this takes depends on the shard count, data volume, recovery concurrency and whether the full shards need to be replaced or not. This need to happen even if if bring up zone C before migrating off zone B but at least this way you always have a replica in place which is less risky. ---
**Author:** ![cadirol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cadirol/32/38459_2.png) [@cadirol](https://discuss.elastic.co/u/cadirol)\ **Post date:** [August 8, 2020, 10:16am UTC](https://discuss.elastic.co/t/multi-node-fail/244244/7 "2020-08-08T10:16:45Z")
Yes this was my tough as well. It was aprox 2000 shard to recover, with at least 1G interconnection. the recover concurrency was set to 8 for incomming and outgoing. but this placed the recovering nodes in to throttled state, which ended up in a deadlock without any progress of recovery. all the nodes did not made any recovery tasks any more, because of throttling state. Which confused me! I think as well, to bring up the zone C is less risky and a good solution. But the CFO thinks different... 🙂 ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 9, 2020, 4:52pm UTC](https://discuss.elastic.co/t/multi-node-fail/244244/8 "2020-08-09T16:52:13Z")
> [@cadirol](#): > > which ended up in a deadlock without any progress of recovery. > all the nodes did not made any recovery tasks any more, because of throttling state. Which confused me! That doesn't make sense to me, throttling will not lead to a deadlock here. Adjusting the concurrent recoveries setting may lead to a deadlock, although you tend to need to set it higher than 8 to hit that issue. I recommend leaving it at the default anyway. 6.4 is long past [the end of its supported life](https://www.elastic.co/support/eol) so you should upgrade for sure. Newer versions have seen improvements in recovery performance. > [@cadirol](#): > > I think as well, to bring up the zone C is less risky and a good solution. But the CFO thinks different... 🙂 The [manual](https://www.elastic.co/guide/en/elasticsearch/reference/current/high-availability-cluster-design.html) for more recent versions covers your questions in some depth. Two zones of data nodes is sufficient, but you do need three for your master-eligible nodes. ---
**Author:** ![cadirol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cadirol/32/38459_2.png) [@cadirol](https://discuss.elastic.co/u/cadirol)\ **Post date:** [August 10, 2020, 12:20pm UTC](https://discuss.elastic.co/t/multi-node-fail/244244/9 "2020-08-10T12:20:14Z")
I think i nailed it down. The issue was network related. Between DC A and DC B was a Ethernet Link with a MTU of 9000 Bytes. On all nodes was a MTU of 9000 set as well. Now, between DC A and DC C was only a MTU of 1500Bytes. The shard replication did fail, because all nodes tried to send the data with a MTU of 9000 Bytes! Now i changed the MTU of all nodes on DC C down to 1500 Bytes - by knowing the downgrade. However, for our amount of data this should be sufficient. After setting the MTU to 1500 Bytes on nodes in DC C - all the replicas are successfully created. As far, Elasticsearch worked as expected Thank you guys for your replies! ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 10, 2020, 12:29pm UTC](https://discuss.elastic.co/t/multi-node-fail/244244/10 "2020-08-10T12:29:06Z")
Thanks for circling back @cadirol that makes much more sense. Shard recoveries send data at quite a high rate so you're more likely to exceed something like an MTU limit during recovery. ---
**Author:** ![Steve\_Mushero](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_mushero/32/22441_2.png) [@Steve\_Mushero](https://discuss.elastic.co/u/Steve_Mushero)\ **Post date:** [August 10, 2020, 2:10pm UTC](https://discuss.elastic.co/t/multi-node-fail/244244/11 "2020-08-10T14:10:32Z")
Can I ask how you found this, as I'd never have thought to check MTU? I guess a good sniff would work, or maybe there's a NIC stat, but likely that'd show up on the link gear, not the node. ---
**Author:** ![cadirol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cadirol/32/38459_2.png) [@cadirol](https://discuss.elastic.co/u/cadirol)\ **Post date:** [August 11, 2020, 8:01am UTC](https://discuss.elastic.co/t/multi-node-fail/244244/12 "2020-08-11T08:01:33Z")
Hi @Steve_Mushero You are right, on ES side you will never get a log message concerning the Network MTU! My way was: I had a index with 3 replica, with two availability zones you will get the primary and one replica on DC A and the two other replicas on DC B. The Shard recovery worked fine for this particular shard on DC A but not to DC B! So, i knew there must be something in between. And as i am a Linux/Elasticsearch/Network Engineer i have access to all related devices. Next step was to check network connectivity on the affected Cisco Nexus Switches, and with "show interface xy" you will see a counter called "jumbo" and "giants". And that was it. I checked the network config on the ES node and found the MTU setting of 9000 Bytes. A ping Test between the nodes with "ping -s 3000 " confirmed me the MTU issue. Now i did set the MTU on all nodes in DC B to 1500 Bytes. And all worked well as expected. It would be a nice feature, if Elasticsearch could detect a MTU issue! But i don't know if it is worth to post a Feature request?! However, Happy Elasticsearching and never forget the MTU of your Network! ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 11, 2020, 8:33am UTC](https://discuss.elastic.co/t/multi-node-fail/244244/13 "2020-08-11T08:33:11Z")
> [@cadirol](#): > > It would be a nice feature, if Elasticsearch could detect a MTU issue! But i don't know if it is worth to post a Feature request?! I don't think we can do that, userspace applications like Elasticsearch don't get to see this sort of detail. All that Elasticsearch sees is that some node-to-node messages don't get through and eventually the connection drops. I wouldn't call this a "deadlock" tho, you would see shard recoveries failing with a `NodeDisconnectedException` or similar and giving up after a few retries. Fortunately a `NodeDisconnectedException` is a clear indication that there's something wrong on the network, and with that clue it shouldn't take long to find the culprit. We recently added [docs](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/system-config-tcpretries.html) that would help you to configure Elasticsearch to fail faster in this situation at least. Incidentally there's nothing inherently wrong with having a mix of MTUs on your network unless it also does something silly like blocking ICMP `fragmentation needed` messages, or you've otherwise hobbled [PMTUD](https://en.wikipedia.org/wiki/Path_MTU_Discovery). Probably worth investigating that more deeply, it'd be better to use jumbo frames where you can. ---
**Author:** ![cadirol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cadirol/32/38459_2.png) [@cadirol](https://discuss.elastic.co/u/cadirol)\ **Post date:** [August 11, 2020, 11:32am UTC](https://discuss.elastic.co/t/multi-node-fail/244244/14 "2020-08-11T11:32:41Z")
> [@DavidTurner](#): > > I don't think we can do that, userspace applications like Elasticsearch don't get to see this sort of detail. All that Elasticsearch sees is that some node-to-node messages don't get through and eventually the connection drops. I wouldn't call this a "deadlock" tho, you would see shard recoveries failing with a `NodeDisconnectedException` or similar and giving up after a few retries. Fortunately a `NodeDisconnectedException` is a clear indication that there's something wrong on the network, and with that clue it shouldn't take long to find the culprit. Thank you for this hint! As i understand right, if `NodeDisconnectedException` in the logs occur, the node is vanished from the cluster inventory. right? i think during our journey with cluster restoring, all node were connected to the cluster! The `TCP retransmission timeout` would be a test worth! thank you! ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 11, 2020, 11:34am UTC](https://discuss.elastic.co/t/multi-node-fail/244244/15 "2020-08-11T11:34:26Z")
> [@cadirol](#): > > As i understand right, if `NodeDisconnectedException` in the logs occur, the node is vanished from the cluster inventory. right? No, that's not right, you can get a `NodeDisconnectedException` without the node leaving the cluster if, for instance, the disconnection is between two data nodes and doesn't involve the master. ---
**Author:** ![cadirol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cadirol/32/38459_2.png) [@cadirol](https://discuss.elastic.co/u/cadirol)\ **Post date:** [August 11, 2020, 11:47am UTC](https://discuss.elastic.co/t/multi-node-fail/244244/16 "2020-08-11T11:47:35Z")
Oh, thank you for this hint! Yes something like `NodeDisconnectedException` would lead to a network issue! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 8, 2020, 11:47am UTC](https://discuss.elastic.co/t/multi-node-fail/244244/17 "2020-09-08T11:47:44Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.