Elasticsearch moving node out of cluster

Elastic Stack Elasticsearch
1

Hi

I have a cluster by name "as_elasticsearch" and I have deleted a node from that cluster as below

curl -XPUT P.P.P.P:9200/_cluster/settings -H 'Content-Type: application/json' -d '{
"transient" :{
"cluster.routing.allocation.exclude._ip" : "X.X.X.X"
}
}';

Then I changed out-of-cluster node elasticsearch.yml as below

path.logs: "/var/opt/novell/nam/logs/elasticsearch/"
path.data: "/var/lib/elasticsearch"

then I assumed that it will be running in default cluster. But its not working as expected

:~ # curl localhost:9200
{
"name" : "",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "na",
"version" : {
"number" : "7.4.2",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "2f90bbf7b93631e52bafb59b3b049cb44ec25e96",
"build_date" : "2019-10-28T20:40:44.881551Z",
"build_snapshot" : false,
"lucene_version" : "8.2.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

If I run

curl localhost:9200/_cat/nodes

{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}

After deleting /var/lib/elasticsearch, its working

NAM-AS1-201:~ # curl localhost:9200/_cat/nodes
127.0.0.1 17 67 3 0.17 0.43 0.58 dilm * --**

Should I do snapshot/restore before deleting data foldler?

2

Can you explain a little more what you are trying to achieve here?

3

I have removed a node out of cluster and was trying to make it work as single node default cluster with default elasticsearch yml

4

Ok, then you need to wipe the data directory entirely. You can't just remove a node from an existing cluster like that and expect it to create it's own cluster, as it still has the cluster state stored.

1 Like
5

Thank you for the reply. So to safeguard my data, do I need to do snapshot/restore??

6

If you want the new cluster to hold data from the main cluster you need to take a snapshot from the main cluster and restore the desired data to the new one once this has been created.

1 Like
7

One more question, I have three nodes in a cluster as-elasticsearch

node-1
node-2
node -3

I want to remove node-2, node-3 nodes as it went down. Then I brought up two new nodes, node-4, node-5. I have deleted /var/lib/elasticsearch folder in both node-4 & node-5 and configured cluster again in node-1, node-4 & node5.
I expected now same cluster will have three nodes node-1, node-4, node-5

But as I see, node-1 has different cluster-id and node-4, node-5 in different cluster-id

Node-1

{
"name" : "node-1",
"cluster_name" : "as_elasticsearch",
"cluster_uuid" : "na",
"version" : {
"number" : "7.4.2",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "2f90bbf7b93631e52bafb59b3b049cb44ec25e96",
"build_date" : "2019-10-28T20:40:44.881551Z",
"build_snapshot" : false,
"lucene_version" : "8.2.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

Node - 4

{
"name" : "node-4",
"cluster_name" : "as_elasticsearch",
"cluster_uuid" : "U5lhso4pQu-RhAsQ00bKpw",
"version" : {
"number" : "7.4.2",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "2f90bbf7b93631e52bafb59b3b049cb44ec25e96",
"build_date" : "2019-10-28T20:40:44.881551Z",
"build_snapshot" : false,
"lucene_version" : "8.2.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"

Why "cluster_uuid" : "na" for node-1??

8

If you lose a majority of master eligible nodes in a 7.x cluster you can not just replace list nodes but rather have to rebuild a cluster from scratch and restore a snapshot. Have a look at the documentation for further details.

9

Is this possible in higher versions? Is this behaviour specific in 7.x cluster?

10

I expect it to apply from 7,0 upwards.

11

Thanks. Even if my two nodes went down in 3-node cluster, can I take snapshot with available node?

curl -X PUT "...:9200/_snapshot/my_backup?pretty" -H 'Content-Type: application/json' -d'
{
"type": "fs",
"settings": {
"location": "/var/"
}
}
'

{
"error" : {
"root_cause" : [
{
"type" : "master_not_discovered_exception",
"reason" : null
}
],
"type" : "master_not_discovered_exception",
"reason" : null
},
"status

It seems like I cant do anything if two
nodes go down.

12

You need a majority of nodes available to take a snapshot. If you do not have an existing snapshot and can not bring one of the list nodes back with all data present I believe you have list the data.

13

I dont have existing snapshot and also I cant bring my two nodes back. So what should I do now?

14

As far as I know there is nothing you can do as the data is lost. This section contains some information.

15

So what do you recommend to avoid such scenarios?

16

Make sure your master eligible nodes do not share hardware or storage and are independent so hardware failure does not result in more than one being lost. Take snapshots on a regular basis to allow you to restore in case of catastrophic failure.

17

Thanks for quick replies. So I will take snapshot everyday. Just one final question, snapshot will take whole snapshot everyday.correct? So I should delete yesterday snapshot and take new snapshot everyday

18

Snapshots in the same repository are incremental.

19

Ok I got it. If I want to take snapshot everyday, I need to create snapshot as
PUT /_snapshot/my_backup/%3Csnapshot-%7Bnow%2Fd%7D%3E with date
everyday. While restoring, should I give same pattern %3Csnapshot-%7Bnow%2Fd%7D%3E so that I will get full data???

20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.