# Recovering a cluster when less than quorum number of master nodes are healthy **URL:** \ **Category:** Elasticsearch\ **Created:** [October 29, 2026, 10:56am UTC](https://discuss.elastic.co/t/recovering-a-cluster-when-less-than-quorum-number-of-master-nodes-are-healthy/391270 "2026-10-29T10:56:40Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Dheeraj\_Gupta](https://avatars.discourse-cdn.com/v4/letter/d/49beb7/32.png) [@Dheeraj\_Gupta](https://discuss.elastic.co/u/Dheeraj_Gupta)\ **Post date:** [October 29, 2026, 10:56am UTC](https://discuss.elastic.co/t/recovering-a-cluster-when-less-than-quorum-number-of-master-nodes-are-healthy/391270/1 "2026-10-29T10:56:40Z")
We have a 9 node self managed cluster running on 3 physical machines. Current version - 9.5.2. There are 3 master only nodes and 6 data nodes. Each physical machine has 1 master + 2 data nodes. Under normal circumstances, we can tolerate failure of any one physical machine (or multiple VMs). Today we had a bad power issue with backup UPS also crashing before cluster could be shutdown safely. On restart, 2 masters refused to start with errors like ```auto org.elasticsearch.ElasticsearchException: Failed to bind service at org.elasticsearch.server@9.5.2/org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:364) at org.elasticsearch.server@9.5.2/org.elasticsearch.node.Node.(Node.java:191) at org.elasticsearch.server@9.5.2/org.elasticsearch.bootstrap.Elasticsearch$1.(Elasticsearch.java:437) at org.elasticsearch.server@9.5.2/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:437) at org.elasticsearch.server@9.5.2/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:102) Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected suffix=5m4b, got=5n2y (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/data/elasticsearch/_state/segments_5m4b"))) ``` One master booted OK but was stuck waiting for quorum to form ```auto [2026-10-29T13:40:59,537][WARN][o.e.c.c.ClusterFormationFailureHelper] [esmaster0] master not discovered or elected yet, an election requires at least 2 nodes with ids from [5SOVytFpTfi72rbFEVrwDw, HPzuaTzgR3KxSSoSGhzbTg, 0Z9YP3dJTxeF5ViW599MPQ], have only discovered non-quorum [{esmaster0}{0Z9YP3dJTxeF5ViW599MPQ}{7Z2tv-aPSFCHehUfWm8VFw}{esmaster0}{10.44.0.43}{10.44.0.43:9201}{m}{9.5.2}{8000099-9111000}]; ``` My understanding is that 2 master disks had inconsistencies due to abrupt power failure and cluster state they had became corrupted. But it seems third node had _some_ form of cluster state on its disk which could have been used to put the cluster into usable state with _maybe some_ data loss. One way would be to lower the quorum so that cluster can be started and then new masters pick up whatever last good state the sole master has. Elasticsearch understandably baulks at this option because it has no way of seeing if master nodes are actually dead or if a network link has broken which may cause "split brain" and destroy cluster state. I found that the only way to have any sort of recovery is to do an `unsafe-bootstrap`on the working master and then try to detach and reattach data nodes to the cluster. When I ran it on the "working" master I got an error ```auto # /usr/share/elasticsearch/bin/elasticsearch-node unsafe-bootstrap WARNING: Elasticsearch MUST be stopped before running this tool. WARNING: A restricted method in java.lang.foreign.Linker has been called WARNING: java.lang.foreign.Linker::downcallHandle has been called by org.apache.lucene.store.PosixNativeAccess in an unnamed module (file:/usr/share/elasticsearch/lib/lucene-core-10.5.1.jar) WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for callers in this module WARNING: Restricted methods will be blocked in a future release unless native access is enabled Oct 29, 2026 2:47:00 PM org.apache.lucene.internal.vectorization.VectorizationProvider lookup WARNING: Java vector incubator module is not readable. For optimal vector performance, pass '--add-modules jdk.incubator.vector' to enable Vector API. org.elasticsearch.gateway.CorruptStateException: java.lang.ClassCastException: class org.elasticsearch.cluster.coordination.ElasticsearchNodeCommand$UnknownProjectCustom cannot be cast to class org.elasticsearch.cluster.metadata.ViewMetadata (org.elasticsearch.cluster.coordination.ElasticsearchNodeCommand$UnknownProjectCustom and org.elasticsearch.cluster.metadata.ViewMetadata are in unnamed module of loader 'app') at org.elasticsearch.gateway.PersistedClusterStateService.readXContent(PersistedClusterStateService.java:702) at org.elasticsearch.gateway.PersistedClusterStateService.lambda$loadOnDiskState$6(PersistedClusterStateService.java:599) at org.elasticsearch.gateway.PersistedClusterStateService.consumeFromType(PersistedClusterStateService.java:745) at org.elasticsearch.gateway.PersistedClusterStateService.loadOnDiskState(PersistedClusterStateService.java:598) at org.elasticsearch.gateway.PersistedClusterStateService.loadBestOnDiskState(PersistedClusterStateService.java:522) at org.elasticsearch.gateway.PersistedClusterStateService.loadBestOnDiskState(PersistedClusterStateService.java:433) at org.elasticsearch.cluster.coordination.ElasticsearchNodeCommand.loadTermAndClusterState(ElasticsearchNodeCommand.java:152) at org.elasticsearch.cluster.coordination.UnsafeBootstrapMasterCommand.processDataPaths(UnsafeBootstrapMasterCommand.java:76) at org.elasticsearch.cluster.coordination.ElasticsearchNodeCommand.processDataPaths(ElasticsearchNodeCommand.java:166) at org.elasticsearch.cluster.coordination.ElasticsearchNodeCommand.execute(ElasticsearchNodeCommand.java:184) at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:55) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:108) at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:96) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:108) at org.elasticsearch.cli.Command.main(Command.java:55) at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:69) Caused by: java.lang.ClassCastException: class org.elasticsearch.cluster.coordination.ElasticsearchNodeCommand$UnknownProjectCustom cannot be cast to class org.elasticsearch.cluster.metadata.ViewMetadata (org.elasticsearch.cluster.coordination.ElasticsearchNodeCommand$UnknownProjectCustom and org.elasticsearch.cluster.metadata.ViewMetadata are in unnamed module of loader 'app') at org.elasticsearch.cluster.metadata.ProjectMetadata$Builder.viewMetadata(ProjectMetadata.java:1512) at org.elasticsearch.cluster.metadata.ProjectMetadata$Builder.build(ProjectMetadata.java:1708) at org.elasticsearch.cluster.metadata.ProjectMetadata$Builder.build(ProjectMetadata.java:1632) at org.elasticsearch.cluster.metadata.ProjectMetadata$Builder.fromXContent(ProjectMetadata.java:2263) at org.elasticsearch.cluster.metadata.Metadata$Builder.readProjects(Metadata.java:1848) at org.elasticsearch.cluster.metadata.Metadata$Builder.fromXContent(Metadata.java:1776) at org.elasticsearch.gateway.PersistedClusterStateService.readXContent(PersistedClusterStateService.java:700) ... 15 more ``` Other commands, like `remove-customs` also failed with same traceback. With no way out (and no snapshots), I bit the bullet and rebuilt the entire cluster , thus losing _all_ data (inc templates, kibana saved searches, dashboards etc.). Based on lessons learnt, I will try to setup snapshot repo so future failures can be recovered safely. However, I have few questions regarding today's events: 1. Was the only working master also corrupt or was this just some bug in `elasticsearch-node` which I could have somehow worked around? 2. As elasticsearch service had started on one master and assuming state on it was not corrupt, is there any other way I can tell cluster to boot up with this one good master? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [October 29, 2026, 1:30pm UTC](https://discuss.elastic.co/t/recovering-a-cluster-when-less-than-quorum-number-of-master-nodes-are-healthy/391270/2 "2026-10-29T13:30:31Z")
`elasticsearch-node` is not failing because it found evidence of corruption. This does not confirm that the data is intact, just that it didn't find a corruption issue before hitting some other problem. Unfortunately it's very much a best-effort tool and it cannot guarantee to recover every cluster. Perhaps we could consider it a bug that it couldn't recover this cluster. There is no other way to safely recover from the loss of 2 of the 3 master-eligible nodes either. Definitely set up snapshots in future, but also note that Elasticsearch is itself designed to be resilient to power failures, as long as your storage hardware performs durable writes correctly (and in particular does not reorder writes across write barriers). To get a `CorruptIndexException` after a power failure indicates that your hardware does not in fact perform durable writes correctly, which you need to address ASAP. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [October 29, 2026, 3:10pm UTC](https://discuss.elastic.co/t/recovering-a-cluster-when-less-than-quorum-number-of-master-nodes-are-healthy/391270/3 "2026-10-29T15:10:12Z")
_Please_ take @DavidTurner ‘s advice. In a 2024 thread you had some translog corruption which David speculated was likely due to your storage hardware? It could be that there are “optimisations” enabled for your storage which make it unsuitable for Elasticsearch, you need check that, but now that you may have been bitten twice it would be my priority to address the storage issues immediately. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [October 29, 2026, 3:29pm UTC](https://discuss.elastic.co/t/recovering-a-cluster-when-less-than-quorum-number-of-master-nodes-are-healthy/391270/4 "2026-10-29T15:29:00Z")
> [@RainTown](#): > > In a 2024 thread Ha well spotted @RainTown[Index stuck in yellow unable to assign replica due to translog corruption - #4 by DavidTurner](https://discuss.elastic.co/t/index-stuck-in-yellow-unable-to-assign-replica-due-to-translog-corruption/356556/4) calls out exactly the same issue: incorrectly reordering `write()` and `fsync()` calls. Your storage is apparently still broken. ---
**Author:** ![Dheeraj\_Gupta](https://avatars.discourse-cdn.com/v4/letter/d/49beb7/32.png) [@Dheeraj\_Gupta](https://discuss.elastic.co/u/Dheeraj_Gupta)\ **Post date:** [October 29, 2026, 3:54pm UTC](https://discuss.elastic.co/t/recovering-a-cluster-when-less-than-quorum-number-of-master-nodes-are-healthy/391270/5 "2026-10-29T15:54:24Z")
Thanks @DavidTurner and @RainTown for weighing in. Can you please provide some pointers regarding what optimizations may be at fault at storage level? This is the first time I've had catastrophic dual node failure even though there have been many unsafe shutdowns over last 8 years the cluster had been running. The physical machine are RAID1 and expose an LVM to the VM which uses virtio and Linix KVM. Physical disk failures have not occurred.