Cluster unable to elect new master

Our data cluster seems to be unable to elect a new master. Due to unfortunate circumstances our master nodes have gone offline and cannot be recovered. (This happened when the cluster was active and didn't impact it at all)

So in order to elect a new master we shut down the cluster and added a new master node to be elected. However upon starting the cluster back up we were greeted with the following message:

[2019-08-01T22:22:12,132][WARN ][o.e.c.c.ClusterFormationFailureHelper] [Data-2] master not discovered or elected yet, an election requires at least 2 nodes with ids from [-z_8zevIRyK_5oT5tWu9ag, a8CTfVusTeKswJgc1b5nKA, X22bHIZUTnSNq6-f7TpUjQ], have discovered [{Master-2}{J9k8SAeiS1CGm2tIp1B0oA}{wjVLrwDgQ6mZ033DRkTTuw}{}{}{ml.machine_memory=67338063872, ml.max_open_jobs=20, xpack.installed=true}, {Data-1}{FkP3072lTKuxQkigN6P3PQ}{IxFscR32SWKDk0upiiQTZQ}{}{}{ml.machine_memory=134772142080, ml.max_open_jobs=20, xpack.installed=true}, {Data-3}{Wd96XK4TTb6mvntqv1X1DA}{Dty9XE-dSl6LChbtcz29Ig}{}{}{ml.machine_memory=134772142080, ml.max_open_jobs=20, xpack.installed=true}] which is not a quorum; discovery will continue using [,,,] from hosts providers and [{Data-2}{_mQO44eCQDyaW89E2hPiaQ}{PufggL7zSe-z2iJt1LmUzg}{}{}{ml.machine_memory=134772142080, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 8, last-accepted version 352 in term 6

It would seem that the cluster is still looking for the old master nodes to start the election. Is there a way around this? We've looked through all the forum posts but couldn't find a solution.

Thank you in advanced.

Your best bet is to restore the whole cluster from a recent snapshot and then index any missing documents again. The master nodes hold information that isn't reliably stored anywhere else, and if that information is gone then there's no other safe way to proceed.

I see thank you!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.