Half master nodes are down, Is there a way to recover cluster?

Elastic Search
1

I had a cluster of several elasticsearch nodes. Four of them where master elegible nodes. We lost two of them, and they're currently in an irrecoverable state.

I removed the lost nodes from the elastic cluster.initial_master_nodes configuration, but myy cluster is unhealthy, every curl command I send gives me back 503 master_not_discovered_exception, and the nodes log show "master not discovered or elected yet, an election requires at least 2 nodes with ids from [...]"

Any Ideas ?

2

Welcome to the forum. I hope someone will be able to help.

There are a number of scenarios covered here:

specifically

Even numbers of master-eligible nodes are often not a great idea. But if you'd had 3 masters, and lost 2, you would be in much the same boat. How many nodes were in your complete cluster? How was the data replicated? Do you have (recent) snapshots?

I'd be very cautious before proceeding. Make a backup of what you've got right now, before trying anything. and wait for a bit as others might respond on the thread. Maybe setup a similar cluster elsewhere, eg AWS, and try to simulate whats happened and practice on that one first.

Good luck!