Elasticsearch cluster health red and data backups

[Roshan_Jha]

I have Elasticsearch cluster of 3 nodes, it was working fine till now then i changed default path.data path to my own custom data path and restarted the whole cluster now my cluster is in red state and when running cluster health api it says master node not elected yet.

so my question is when we change the data.path does elastic node chnages it id too because in log i can see one of my node cannot find other two nodes i am sharing the log for clarification.

node 1

[2020-05-28T17:08:53,938][WARN ][o.e.c.c.ClusterFormationFailureHelper] [iamesd3_upg] master not discovered or elected yet, an election requires at least 2 nodes with ids from [6LTiaYAHTrSEdbvPPtlzmg, JfD9-6IwRIGXmZUkepUzHg, 8LHaAmg4SGKnlAxnN-IJSQ], have discovered [{iamesd2_upg}{VAWn0Pm4R_-ITKptSu4MCg}{hP-a34WMQdeJUJmE3Wydxw}{10.72.21.41}{10.72.21.41:9300}{ml.machine_memory=8192020480, ml.max_open_jobs=20, xpack.installed=true}, {iamesd_upg}{cA8K1RFYRP6rtkeIVSB20Q}{qGmIhscERPehh40Rq8s7mQ}{10.72.21.40}{10.72.21.40:9300}{ml.machine_memory=8191987712, ml.max_open_jobs=20, xpack.installed=true}] which is not a quorum; discovery will continue using [10.72.21.40:9300, 10.72.21.41:9300] from hosts providers and [{iamesd3_upg}{8LHaAmg4SGKnlAxnN-IJSQ}{MpUwSSXiTbGfduXqufhlTQ}{10.72.21.42}{10.72.21.42:9300}{ml.machine_memory=8191995904, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 2614, last-accepted version 5868 in term 2614

node2

[2020-05-28T17:19:56,276][WARN ][o.e.c.c.ClusterFormationFailureHelper] [iamesd_upg] master not discovered or elected yet, an election requires at least 2 nodes with ids from [cA8K1RFYRP6rtkeIVSB20Q, VAWn0Pm4R_-ITKptSu4MCg, 8LHaAmg4SGKnlAxnN-IJSQ], have discovered [{iamesd3_upg}{8LHaAmg4SGKnlAxnN-IJSQ}{OacA1YSnT_egR_fBjL1wRw}{10.72.21.42}{10.72.21.42:9300}{ml.machine_memory=8191995904, ml.max_open_jobs=20, xpack.installed=true}, {iamesd2_upg}{VAWn0Pm4R_-ITKptSu4MCg}{vtoy2OBgSJ-pr8qBfkBVZA}{10.72.21.41}{10.72.21.41:9300}{ml.machine_memory=8192020480, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [10.72.21.41:9300, 10.72.21.42:9300] from hosts providers and [{iamesd_upg}{cA8K1RFYRP6rtkeIVSB20Q}{qGmIhscERPehh40Rq8s7mQ}{10.72.21.40}{10.72.21.40:9300}{ml.machine_memory=8191987712, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 2614, last-accepted version 5686 in term 2614

node3

[2020-05-28T17:19:37,411][WARN ][o.e.c.c.ClusterFormationFailureHelper] [iamesd2_upg] master not discovered or elected yet, an election requires at least 2 nodes with ids from [VAWn0Pm4R_-ITKptSu4MCg, 6LTiaYAHTrSEdbvPPtlzmg, 8LHaAmg4SGKnlAxnN-IJSQ], have discovered [{iamesd_upg}{cA8K1RFYRP6rtkeIVSB20Q}{qGmIhscERPehh40Rq8s7mQ}{10.72.21.40}{10.72.21.40:9300}{ml.machine_memory=8191987712, ml.max_open_jobs=20, xpack.installed=true}, {iamesd3_upg}{8LHaAmg4SGKnlAxnN-IJSQ}{OacA1YSnT_egR_fBjL1wRw}{10.72.21.42}{10.72.21.42:9300}{ml.machine_memory=8191995904, ml.max_open_jobs=20, xpack.installed=true}] which is not a quorum; discovery will continue using [10.72.21.40:9300, 10.72.21.42:9300] from hosts providers and [{iamesd2_upg}{VAWn0Pm4R_-ITKptSu4MCg}{vtoy2OBgSJ-pr8qBfkBVZA}{10.72.21.41}{10.72.21.41:9300}{ml.machine_memory=8192020480, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 2614, last-accepted version 5752 in term 2614

I cannot solve this issue.

I tried different techniques

1. I copied nodes folder default location and created new folder and pasted nodes folder there.
2. creating the new cluster and pointing new cluster data.path to pasted location.
3. then creating new cluster but the result is same master not discoverable, i think elastic stores nodes information in nodes folder so it can do indexing and all.

for now i am running new elastic cluster but i need that data into cluster anyhow it's important for client.

one more thing can we change node-id if yes this problem might be solved.

please help me out with this issue.

[warkolm (Mark Walkom)]

If you change the path.data then you're essentially reinitialising a new Elasticsearch node. No data, nothing.

You might be able to get it to work if you stop the node(s), change that value, copy the old path.data contents into the new one, and then restart the nodes.

[Christian_Dahlqvist]

I think it would help if you outlines exactly which steps you took and what changed at which point as well as which version of Elasticsearch you are using.

[Roshan_Jha]

elastic version is 7.2.0

we are using tar of elastic.

steps:- copy data folder from elastic default location to my custom location and set the same path in data.path in elasticsearch.yml file.
then we started the elastic cluster then we got the warning as master not discovered full log is described in above message.

as above did not work we extracted the new elastic and used new cluster but we need the old cluster data anyhow.

[Roshan_Jha]

i tried like below, i think its same as you are saying.

we took down the cluster then copied the default nodes folder to new location and then started each node but getting error, master not discoverable, i think it is considering old node id and when configuring new node path basically we are creating new nodeid and it does not match.

let me know if i am right.

[Roshan_Jha]

Hi,

actually i solved the problem by trying below steps.

never try below steps, try only if you do not have option because you might loose some data from cluster

1. stop all the elasticservices.
2. used the elasticsearch-node command line which is already present search for this term elasticsearch-node you will come to know but on your own risk.
3. using above tool and botstrap command from above tool i deatched the node from cluster and started the cluster and then rejoined the node in cluster.
4. it was working fine then.