# Elasticsearch moving node out of cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [November 8, 2020, 6:58am UTC](https://discuss.elastic.co/t/elasticsearch-moving-node-out-of-cluster/254653 "2020-11-08T06:58:03Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [November 8, 2020, 6:58am UTC](https://discuss.elastic.co/t/elasticsearch-moving-node-out-of-cluster/254653/1 "2020-11-08T06:58:03Z")
Hi I have a cluster by name "as\_elasticsearch" and I have deleted a node from that cluster as below curl -XPUT P.P.P.P:9200/\_cluster/settings -H 'Content-Type: application/json' -d '{ "transient" :{ "cluster.routing.allocation.exclude.\_ip" : "X.X.X.X" } }'; Then I changed out-of-cluster node elasticsearch.yml as below path.logs: "/var/opt/novell/nam/logs/elasticsearch/" path.data: "/var/lib/elasticsearch" then I assumed that it will be running in default cluster. But its not working as expected **:~ # curl localhost:9200 { "name" : "**", "cluster\_name" : "elasticsearch", "cluster\_uuid" : "_na_", "version" : { "number" : "7.4.2", "build\_flavor" : "default", "build\_type" : "rpm", "build\_hash" : "2f90bbf7b93631e52bafb59b3b049cb44ec25e96", "build\_date" : "2019-10-28T20:40:44.881551Z", "build\_snapshot" : false, "lucene\_version" : "8.2.0", "minimum\_wire\_compatibility\_version" : "6.8.0", "minimum\_index\_compatibility\_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search" } If I run curl localhost:9200/\_cat/nodes {"error":{"root\_cause":[{"type":"master\_not\_discovered\_exception","reason":null}],"type":"master\_not\_discovered\_exception","reason":null},"status":503} After deleting /var/lib/elasticsearch, its working NAM-AS1-201:~ # curl localhost:9200/\_cat/nodes 127.0.0.1 17 67 3 0.17 0.43 0.58 dilm \* **-** -\*\* Should I do snapshot/restore before deleting data foldler? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 8, 2020, 10:45pm UTC](https://discuss.elastic.co/t/elasticsearch-moving-node-out-of-cluster/254653/2 "2020-11-08T22:45:10Z")
Can you explain a little more what you are trying to achieve here? ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [November 9, 2020, 5:44am UTC](https://discuss.elastic.co/t/elasticsearch-moving-node-out-of-cluster/254653/3 "2020-11-09T05:44:22Z")
I have removed a node out of cluster and was trying to make it work as single node default cluster with default elasticsearch yml ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 9, 2020, 5:49am UTC](https://discuss.elastic.co/t/elasticsearch-moving-node-out-of-cluster/254653/4 "2020-11-09T05:49:39Z")
Ok, then you need to wipe the data directory entirely. You can't just remove a node from an existing cluster like that and expect it to create it's own cluster, as it still has the cluster state stored. ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [November 9, 2020, 6:19am UTC](https://discuss.elastic.co/t/elasticsearch-moving-node-out-of-cluster/254653/5 "2020-11-09T06:19:29Z")
Thank you for the reply. So to safeguard my data, do I need to do snapshot/restore?? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 9, 2020, 6:40am UTC](https://discuss.elastic.co/t/elasticsearch-moving-node-out-of-cluster/254653/6 "2020-11-09T06:40:48Z")
If you want the new cluster to hold data from the main cluster you need to take a snapshot from the main cluster and restore the desired data to the new one once this has been created. ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [November 9, 2020, 2:50pm UTC](https://discuss.elastic.co/t/elasticsearch-moving-node-out-of-cluster/254653/7 "2020-11-09T14:50:52Z")
One more question, I have three nodes in a cluster as-elasticsearch node-1 node-2 node -3 I want to remove node-2, node-3 nodes as it went down. Then I brought up two new nodes, node-4, node-5. I have deleted /var/lib/elasticsearch folder in both node-4 & node-5 and configured cluster again in node-1, node-4 & node5. I expected now same cluster will have three nodes node-1, node-4, node-5 But as I see, node-1 has different cluster-id and node-4, node-5 in different cluster-id Node-1 { "name" : "node-1", "cluster\_name" : "as\_elasticsearch", "cluster\_uuid" : "_na_", "version" : { "number" : "7.4.2", "build\_flavor" : "default", "build\_type" : "rpm", "build\_hash" : "2f90bbf7b93631e52bafb59b3b049cb44ec25e96", "build\_date" : "2019-10-28T20:40:44.881551Z", "build\_snapshot" : false, "lucene\_version" : "8.2.0", "minimum\_wire\_compatibility\_version" : "6.8.0", "minimum\_index\_compatibility\_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search" } Node - 4 { "name" : "node-4", "cluster\_name" : "as\_elasticsearch", "cluster\_uuid" : "U5lhso4pQu-RhAsQ00bKpw", "version" : { "number" : "7.4.2", "build\_flavor" : "default", "build\_type" : "rpm", "build\_hash" : "2f90bbf7b93631e52bafb59b3b049cb44ec25e96", "build\_date" : "2019-10-28T20:40:44.881551Z", "build\_snapshot" : false, "lucene\_version" : "8.2.0", "minimum\_wire\_compatibility\_version" : "6.8.0", "minimum\_index\_compatibility\_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search" Why "cluster\_uuid" : "_na_" for node-1?? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 9, 2020, 3:10pm UTC](https://discuss.elastic.co/t/elasticsearch-moving-node-out-of-cluster/254653/8 "2020-11-09T15:10:25Z")
If you lose a majority of master eligible nodes in a 7.x cluster you can not just replace list nodes but rather have to rebuild a cluster from scratch and restore a snapshot. Have a look at [the documentation](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/high-availability-cluster-design.html) for further details. ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [November 9, 2020, 3:47pm UTC](https://discuss.elastic.co/t/elasticsearch-moving-node-out-of-cluster/254653/9 "2020-11-09T15:47:06Z")
Is this possible in higher versions? Is this behaviour specific in 7.x cluster? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 9, 2020, 3:50pm UTC](https://discuss.elastic.co/t/elasticsearch-moving-node-out-of-cluster/254653/10 "2020-11-09T15:50:40Z")
I expect it to apply from 7,0 upwards. ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [November 10, 2020, 1:15am UTC](https://discuss.elastic.co/t/elasticsearch-moving-node-out-of-cluster/254653/11 "2020-11-10T01:15:56Z")
Thanks. Even if my two nodes went down in 3-node cluster, can I take snapshot with available node? curl -X PUT " **.**. **.** :9200/\_snapshot/my\_backup?pretty" -H 'Content-Type: application/json' -d' { "type": "fs", "settings": { "location": "/var/" } } ' { "error" : { "root\_cause" : [ { "type" : "master\_not\_discovered\_exception", "reason" : null } ], "type" : "master\_not\_discovered\_exception", "reason" : null }, "status It seems like I cant do anything if two nodes go down. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 10, 2020, 6:09am UTC](https://discuss.elastic.co/t/elasticsearch-moving-node-out-of-cluster/254653/12 "2020-11-10T06:09:25Z")
You need a majority of nodes available to take a snapshot. If you do not have an existing snapshot and can not bring one of the list nodes back with all data present I believe you have list the data. ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [November 10, 2020, 6:19am UTC](https://discuss.elastic.co/t/elasticsearch-moving-node-out-of-cluster/254653/13 "2020-11-10T06:19:24Z")
I dont have existing snapshot and also I cant bring my two nodes back. So what should I do now? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 10, 2020, 6:33am UTC](https://discuss.elastic.co/t/elasticsearch-moving-node-out-of-cluster/254653/14 "2020-11-10T06:33:01Z")
As far as I know there is nothing you can do as the data is lost. [This section](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-voting.html) contains some information. ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [November 10, 2020, 6:40am UTC](https://discuss.elastic.co/t/elasticsearch-moving-node-out-of-cluster/254653/15 "2020-11-10T06:40:15Z")
So what do you recommend to avoid such scenarios? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 10, 2020, 6:42am UTC](https://discuss.elastic.co/t/elasticsearch-moving-node-out-of-cluster/254653/16 "2020-11-10T06:42:47Z")
Make sure your master eligible nodes do not share hardware or storage and are independent so hardware failure does not result in more than one being lost. Take snapshots on a regular basis to allow you to restore in case of catastrophic failure. ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [November 10, 2020, 6:56am UTC](https://discuss.elastic.co/t/elasticsearch-moving-node-out-of-cluster/254653/17 "2020-11-10T06:56:52Z")
Thanks for quick replies. So I will take snapshot everyday. Just one final question, snapshot will take whole snapshot everyday.correct? So I should delete yesterday snapshot and take new snapshot everyday ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 10, 2020, 6:59am UTC](https://discuss.elastic.co/t/elasticsearch-moving-node-out-of-cluster/254653/18 "2020-11-10T06:59:51Z")
Snapshots in the same repository are incremental. ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [November 10, 2020, 7:13am UTC](https://discuss.elastic.co/t/elasticsearch-moving-node-out-of-cluster/254653/19 "2020-11-10T07:13:08Z")
Ok I got it. If I want to take snapshot everyday, I need to create snapshot as PUT /\_snapshot/my\_backup/%3Csnapshot-%7Bnow%2Fd%7D%3E with date everyday. While restoring, should I give same pattern %3Csnapshot-%7Bnow%2Fd%7D%3E so that I will get full data??? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 8, 2020, 7:13am UTC](https://discuss.elastic.co/t/elasticsearch-moving-node-out-of-cluster/254653/20 "2020-12-08T07:13:12Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.