# Managing loss of master quorum from simultaneous restart of nodes
**URL:** https://discuss.elastic.co/t/managing-loss-of-master-quorum-from-simultaneous-restart-of-nodes/225112
**Category:** Elasticsearch
**Created:** [March 26, 2020, 5:13am UTC](https://discuss.elastic.co/t/managing-loss-of-master-quorum-from-simultaneous-restart-of-nodes/225112 "2020-03-26T05:13:11Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@Dandy](https://discuss.elastic.co/u/Dandy)
#### Post date: [March 26, 2020, 5:13am UTC](https://discuss.elastic.co/t/managing-loss-of-master-quorum-from-simultaneous-restart-of-nodes/225112/1 "2020-03-26T05:13:12Z")
Similar in nature to this:
> [@Handling loss of master quorum in ES7+](https://discuss.elastic.co/t/handling-loss-of-master-quorum-in-es7/191443):
>
> I am planning to upgrade an ES 6.3 cluster to 7.2 soon. In ES 6.3 if we lose a quorum of master nodes we can bring up new nodes which will join the cluster and regain quorum, but after ES 7 new master nodes cannot join a cluster which does not have quorum and the cluster will thus be permanently impaired. While we never kill more than one node at a time, we would still like to handle a loss of master quorum due to any unforeseen circumstances or hardware issues, and without bringing up a new clu…
In 7.X I have a scenario where the quorum is lost due to a restart of all the nodes at the same time. All the data is still present, so cluster state is OK.
What's the solution to working around this? I have masters constantly trying to connect to each other but failing:
```auto
[2020-03-26T05:08:41,583][INFO][o.e.c.c.JoinHelper] [elasticsearch-es-master-1] failed to join {elasticsearch-es-master-1}{KG71sIO_TAOPHDCnwVdxRw}{8r3SI6OlRPKF3jfr5A81zw}{10.244.21.61}{10.244.21.61:9300}{box_type=hot} with JoinRequest{sourceNode={elasticsearch-es-master-1}{KG71sIO_TAOPHDCnwVdxRw}{8r3SI6OlRPKF3jfr5A81zw}{10.244.21.61}{10.244.21.61:9300}{box_type=hot}, optionalJoin=Optional[Join{term=94, lastAcceptedTerm=93, lastAcceptedVersion=5099765, sourceNode={elasticsearch-es-master-1}{KG71sIO_TAOPHDCnwVdxRw}{8r3SI6OlRPKF3jfr5A81zw}{10.244.21.61}{10.244.21.61:9300}{box_type=hot}, targetNode={elasticsearch-es-master-1}{KG71sIO_TAOPHDCnwVdxRw}{8r3SI6OlRPKF3jfr5A81zw}{10.244.21.61}{10.244.21.61:9300}{box_type=hot}}]}
org.elasticsearch.transport.RemoteTransportException: [elasticsearch-es-master-1][10.244.21.61:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: node is no longer master for term 95 while handling publication
at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1012) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:252) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:238) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.2.0.jar:7.2.0]
```
It seems they can't establish quorum and appear to be all over the place right now.
---
### Author:  [@Dandy](https://discuss.elastic.co/u/Dandy)
#### Post date: [March 26, 2020, 7:18am UTC](https://discuss.elastic.co/t/managing-loss-of-master-quorum-from-simultaneous-restart-of-nodes/225112/2 "2020-03-26T07:18:29Z")
Okay, looks like it's literally just because we have slow disks and the join/publish timeout is being exceeded constantly. Increased the `cluster.join.timeout` to about 300 seconds and that solved this.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 26, 2020, 7:23am UTC](https://discuss.elastic.co/t/managing-loss-of-master-quorum-from-simultaneous-restart-of-nodes/225112/3 "2020-03-26T07:23:25Z")
If that is the case I would recommend using more performance storage in order to improve stability and availability. What kind of storage are you using?
---
### Author:  [@Dandy](https://discuss.elastic.co/u/Dandy)
#### Post date: [March 26, 2020, 7:58am UTC](https://discuss.elastic.co/t/managing-loss-of-master-quorum-from-simultaneous-restart-of-nodes/225112/4 "2020-03-26T07:58:43Z")
I don't remember exactly what HDD we're using because we use a block storage platform, that being said, we have SSD space so I can migrate the masters to SSDs because it's just sensible.
---
### Author:  [@rcowart](https://discuss.elastic.co/u/rcowart)
#### Post date: [March 26, 2020, 9:52am UTC](https://discuss.elastic.co/t/managing-loss-of-master-quorum-from-simultaneous-restart-of-nodes/225112/5 "2020-03-26T09:52:45Z")
@Dandy you may be interested in this video where I compare various storage options for Elasticsearch...
I tested 1xNVMe, 1xSSD, 2xSSD (RAID-0), 1xHDD, 4xHDD (RAID-0)
**TL;DR** - HDDs are _HORRIBLE_ (even local multi-spindle RAID-optimized). NVMe isn't the best option like you might expect. Multi-SSD (RAID-0 - SATA/SAS) is the way to go.
[](https://www.youtube.com/watch?v=nKUpfJCBiS4)
Rob
[](https://github.com/robcowart) [](https://www.youtube.com/channel/UCivWvTx1DwrWNcDLV58kmOg) [](https://www.linkedin.com/in/robertcowart/)
**[How to install Elasticsearch & Kibana on Ubuntu - incl. hardware recommendations](https://www.youtube.com/watch?v=gZb7HpVOges)**
**[What is the best storage technology for Elasticsearch?](https://www.youtube.com/watch?v=nKUpfJCBiS4)**
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 26, 2020, 10:49am UTC](https://discuss.elastic.co/t/managing-loss-of-master-quorum-from-simultaneous-restart-of-nodes/225112/6 "2020-03-26T10:49:12Z")
Might also be worth upgrading to 7.6 or later since there's been some recent reductions in the IO needs of master-eligible nodes that should make them work much better on slower disks.
---
### Author:  [@Dandy](https://discuss.elastic.co/u/Dandy)
#### Post date: [April 21, 2020, 4:51am UTC](https://discuss.elastic.co/t/managing-loss-of-master-quorum-from-simultaneous-restart-of-nodes/225112/7 "2020-04-21T04:51:14Z")
Thank you @rcowart. Sorry for the delay in responding, this is very much appreciated,
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 19, 2020, 4:51am UTC](https://discuss.elastic.co/t/managing-loss-of-master-quorum-from-simultaneous-restart-of-nodes/225112/8 "2020-05-19T04:51:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.