Recovering a cluster when less than quorum number of master nodes are healthy

Elastic Stack Elasticsearch
1

We have a 9 node self managed cluster running on 3 physical machines. Current version - 9.5.2. There are 3 master only nodes and 6 data nodes. Each physical machine has 1 master + 2 data nodes. Under normal circumstances, we can tolerate failure of any one physical machine (or multiple VMs).

Today we had a bad power issue with backup UPS also crashing before cluster could be shutdown safely. On restart, 2 masters refused to start with errors like


org.elasticsearch.ElasticsearchException: Failed to bind service
	at org.elasticsearch.server@9.5.2/org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:364)
	at org.elasticsearch.server@9.5.2/org.elasticsearch.node.Node.<init>(Node.java:191)
	at org.elasticsearch.server@9.5.2/org.elasticsearch.bootstrap.Elasticsearch$1.<init>(Elasticsearch.java:437)
	at org.elasticsearch.server@9.5.2/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:437)
	at org.elasticsearch.server@9.5.2/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:102)
Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected suffix=5m4b, got=5n2y (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/data/elasticsearch/_state/segments_5m4b")))

One master booted OK but was stuck waiting for quorum to form

[2026-10-29T13:40:59,537][WARN ][o.e.c.c.ClusterFormationFailureHelper] [esmaster0] master not discovered or elected yet, an election requires at least 2 nodes with ids from [5SOVytFpTfi72rbFEVrwDw, HPzuaTzgR3KxSSoSGhzbTg, 0Z9YP3dJTxeF5ViW599MPQ], have only discovered non-quorum [{esmaster0}{0Z9YP3dJTxeF5ViW599MPQ}{7Z2tv-aPSFCHehUfWm8VFw}{esmaster0}{10.44.0.43}{10.44.0.43:9201}{m}{9.5.2}{8000099-9111000}];

My understanding is that 2 master disks had inconsistencies due to abrupt power failure and cluster state they had became corrupted. But it seems third node had some form of cluster state on its disk which could have been used to put the cluster into usable state with maybe some data loss.

One way would be to lower the quorum so that cluster can be started and then new masters pick up whatever last good state the sole master has. Elasticsearch understandably baulks at this option because it has no way of seeing if master nodes are actually dead or if a network link has broken which may cause "split brain" and destroy cluster state.

I found that the only way to have any sort of recovery is to do an unsafe-bootstrapon the working master and then try to detach and reattach data nodes to the cluster.

When I ran it on the "working" master I got an error

# /usr/share/elasticsearch/bin/elasticsearch-node unsafe-bootstrap
WARNING: Elasticsearch MUST be stopped before running this tool.

WARNING: A restricted method in java.lang.foreign.Linker has been called
WARNING: java.lang.foreign.Linker::downcallHandle has been called by org.apache.lucene.store.PosixNativeAccess in an unnamed module (file:/usr/share/elasticsearch/lib/lucene-core-10.5.1.jar)
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for callers in this module
WARNING: Restricted methods will be blocked in a future release unless native access is enabled

Oct 29, 2026 2:47:00 PM org.apache.lucene.internal.vectorization.VectorizationProvider lookup
WARNING: Java vector incubator module is not readable. For optimal vector performance, pass '--add-modules jdk.incubator.vector' to enable Vector API.
org.elasticsearch.gateway.CorruptStateException: java.lang.ClassCastException: class org.elasticsearch.cluster.coordination.ElasticsearchNodeCommand$UnknownProjectCustom cannot be cast to class org.elasticsearch.cluster.metadata.ViewMetadata (org.elasticsearch.cluster.coordination.ElasticsearchNodeCommand$UnknownProjectCustom and org.elasticsearch.cluster.metadata.ViewMetadata are in unnamed module of loader 'app')
	at org.elasticsearch.gateway.PersistedClusterStateService.readXContent(PersistedClusterStateService.java:702)
	at org.elasticsearch.gateway.PersistedClusterStateService.lambda$loadOnDiskState$6(PersistedClusterStateService.java:599)
	at org.elasticsearch.gateway.PersistedClusterStateService.consumeFromType(PersistedClusterStateService.java:745)
	at org.elasticsearch.gateway.PersistedClusterStateService.loadOnDiskState(PersistedClusterStateService.java:598)
	at org.elasticsearch.gateway.PersistedClusterStateService.loadBestOnDiskState(PersistedClusterStateService.java:522)
	at org.elasticsearch.gateway.PersistedClusterStateService.loadBestOnDiskState(PersistedClusterStateService.java:433)
	at org.elasticsearch.cluster.coordination.ElasticsearchNodeCommand.loadTermAndClusterState(ElasticsearchNodeCommand.java:152)
	at org.elasticsearch.cluster.coordination.UnsafeBootstrapMasterCommand.processDataPaths(UnsafeBootstrapMasterCommand.java:76)
	at org.elasticsearch.cluster.coordination.ElasticsearchNodeCommand.processDataPaths(ElasticsearchNodeCommand.java:166)
	at org.elasticsearch.cluster.coordination.ElasticsearchNodeCommand.execute(ElasticsearchNodeCommand.java:184)
	at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:55)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:108)
	at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:96)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:108)
	at org.elasticsearch.cli.Command.main(Command.java:55)
	at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:69)
Caused by: java.lang.ClassCastException: class org.elasticsearch.cluster.coordination.ElasticsearchNodeCommand$UnknownProjectCustom cannot be cast to class org.elasticsearch.cluster.metadata.ViewMetadata (org.elasticsearch.cluster.coordination.ElasticsearchNodeCommand$UnknownProjectCustom and org.elasticsearch.cluster.metadata.ViewMetadata are in unnamed module of loader 'app')
	at org.elasticsearch.cluster.metadata.ProjectMetadata$Builder.viewMetadata(ProjectMetadata.java:1512)
	at org.elasticsearch.cluster.metadata.ProjectMetadata$Builder.build(ProjectMetadata.java:1708)
	at org.elasticsearch.cluster.metadata.ProjectMetadata$Builder.build(ProjectMetadata.java:1632)
	at org.elasticsearch.cluster.metadata.ProjectMetadata$Builder.fromXContent(ProjectMetadata.java:2263)
	at org.elasticsearch.cluster.metadata.Metadata$Builder.readProjects(Metadata.java:1848)
	at org.elasticsearch.cluster.metadata.Metadata$Builder.fromXContent(Metadata.java:1776)
	at org.elasticsearch.gateway.PersistedClusterStateService.readXContent(PersistedClusterStateService.java:700)
	... 15 more

Other commands, like remove-customs also failed with same traceback.

With no way out (and no snapshots), I bit the bullet and rebuilt the entire cluster , thus losing all data (inc templates, kibana saved searches, dashboards etc.). Based on lessons learnt, I will try to setup snapshot repo so future failures can be recovered safely. However, I have few questions regarding today's events:

  1. Was the only working master also corrupt or was this just some bug in elasticsearch-node which I could have somehow worked around?
  2. As elasticsearch service had started on one master and assuming state on it was not corrupt, is there any other way I can tell cluster to boot up with this one good master?
2

elasticsearch-node is not failing because it found evidence of corruption. This does not confirm that the data is intact, just that it didn't find a corruption issue before hitting some other problem. Unfortunately it's very much a best-effort tool and it cannot guarantee to recover every cluster. Perhaps we could consider it a bug that it couldn't recover this cluster.

There is no other way to safely recover from the loss of 2 of the 3 master-eligible nodes either.

Definitely set up snapshots in future, but also note that Elasticsearch is itself designed to be resilient to power failures, as long as your storage hardware performs durable writes correctly (and in particular does not reorder writes across write barriers). To get a CorruptIndexException after a power failure indicates that your hardware does not in fact perform durable writes correctly, which you need to address ASAP.

3

Please take @DavidTurner ā€˜s advice.

In a 2024 thread you had some translog corruption which David speculated was likely due to your storage hardware? It could be that there are ā€œoptimisationsā€ enabled for your storage which make it unsuitable for Elasticsearch, you need check that, but now that you may have been bitten twice it would be my priority to address the storage issues immediately.

4

Ha well spotted @RainTown Index stuck in yellow unable to assign replica due to translog corruption - #4 by DavidTurner calls out exactly the same issue: incorrectly reordering write() and fsync() calls. Your storage is apparently still broken.

5

Thanks @DavidTurner and @RainTown for weighing in.

Can you please provide some pointers regarding what optimizations may be at fault at storage level? This is the first time I've had catastrophic dual node failure even though there have been many unsafe shutdowns over last 8 years the cluster had been running.

The physical machine are RAID1 and expose an LVM to the VM which uses virtio and Linix KVM. Physical disk failures have not occurred.