Our data cluster seems to be unable to elect a new master. Due to unfortunate circumstances our master nodes have gone offline and cannot be recovered. (This happened when the cluster was active and didn't impact it at all)
So in order to elect a new master we shut down the cluster and added a new master node to be elected. However upon starting the cluster back up we were greeted with the following message:
[2019-08-01T22:22:12,132][WARN ][o.e.c.c.ClusterFormationFailureHelper] [Data-2] master not discovered or elected yet, an election requires at least 2 nodes with ids from [-z_8zevIRyK_5oT5tWu9ag, a8CTfVusTeKswJgc1b5nKA, X22bHIZUTnSNq6-f7TpUjQ], have discovered [{Master-2}{J9k8SAeiS1CGm2tIp1B0oA}{wjVLrwDgQ6mZ033DRkTTuw}{192.168.100.2}{192.168.100.2:9300}{ml.machine_memory=67338063872, ml.max_open_jobs=20, xpack.installed=true}, {Data-1}{FkP3072lTKuxQkigN6P3PQ}{IxFscR32SWKDk0upiiQTZQ}{192.168.100.3}{192.168.100.3:9300}{ml.machine_memory=134772142080, ml.max_open_jobs=20, xpack.installed=true}, {Data-3}{Wd96XK4TTb6mvntqv1X1DA}{Dty9XE-dSl6LChbtcz29Ig}{192.168.100.5}{192.168.100.5:9300}{ml.machine_memory=134772142080, ml.max_open_jobs=20, xpack.installed=true}] which is not a quorum; discovery will continue using [192.168.100.1:9300, 192.168.100.2:9300, 192.168.100.3:9300, 192.168.100.5:9300] from hosts providers and [{Data-2}{_mQO44eCQDyaW89E2hPiaQ}{PufggL7zSe-z2iJt1LmUzg}{192.168.100.4}{192.168.100.4:9300}{ml.machine_memory=134772142080, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 8, last-accepted version 352 in term 6
It would seem that the cluster is still looking for the old master nodes to start the election. Is there a way around this? We've looked through all the forum posts but couldn't find a solution.
Thank you in advanced.