# ElasticSearch cluster restarting/adding node(s) **URL:** \ **Category:** Elasticsearch\ **Created:** [March 2, 2018, 12:30pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-restarting-adding-node-s/122249 "2018-03-02T12:30:33Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![szymon](https://avatars.discourse-cdn.com/v4/letter/s/34f0e0/32.png) [@szymon](https://discuss.elastic.co/u/szymon)\ **Post date:** [March 2, 2018, 12:30pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-restarting-adding-node-s/122249/1 "2018-03-02T12:30:33Z")
Hello, i have some questions about how ES should behave while restarting or adding nodes. CASE 1: I have 4 nodes in a cluster (every node can be elected as a master and also is a data node). Number of replicas is set to 0. Minimum\_master\_nodes is set to 3. Let's say I restart one of the nodes (which hold partition\_1 data). While one node is restarting, rest of the nodes will assume that partition\_1 is gone. Question 1.1: Is it possible that rest of nodes will create new partition\_1 and after rejoining of restarted node, original/new data from partition\_1 will be lost? Question 1.2: Just to be sure: Let's say restarted node cannot join back to the cluster. So will it reject 'write' operations but will allow 'read' operations? CASE 2: I have single-node cluster . I'd like to add a second node. So i change minimum\_master\_nodes to 2 and start second node with the same configuration (every node can be elected as a master). Question 2.1: Is it possible that the second (new) node will be elected as a master - if so, what will happen with the data from first (old) node? Will the data be lost/splitted beetwen nodes? Thank you in advance, Szymon ---
**Author:** ![Ben96](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ben96/32/28334_2.png) [@Ben96](https://discuss.elastic.co/u/Ben96)\ **Post date:** [March 2, 2018, 12:48pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-restarting-adding-node-s/122249/2 "2018-03-02T12:48:45Z")
Hi, Case 1 - If one of your 4 Nodes is restarted via systemctl/reboot, Elastic will not be able to assume the missing partition\_1. You shall see "Unassigned Shard(s)" in this case. To solve your case, you need to `EXLUDE` node before restart, in this case, remaining nodes will assume your "partition\_1" by reallocating shards correctly between alive nodes Example for excluding : ``` curl --user elastic:changeme -XPUT ':9200/_cluster/settings' -d '{ "transient" :{ "cluster.routing.allocation.exclude._name" : "" } }' ``` Case 1 Bis - Yes, you will need (if you do my solution) to include the node, else if your are restarting node, elastic will reassigned missing shards to the right node before the restart . Case 2 : Yes, if you configured the node 2 to be eligible as Master Node and no, no change will appears the your data. If all this isn't clear for you, tell me, i would take more times to answer with more details ---
**Author:** ![szymon](https://avatars.discourse-cdn.com/v4/letter/s/34f0e0/32.png) [@szymon](https://discuss.elastic.co/u/szymon)\ **Post date:** [March 5, 2018, 7:58am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-restarting-adding-node-s/122249/3 "2018-03-05T07:58:10Z")
Thank you for answer. I have some follow-up questions regarding case 1: Question 1.4: > [@Ben96](#): > > Case 1 - If one of your 4 Nodes is restarted via systemctl/reboot, Elastic will not be able to assume the missing partition\_1. You shall see "Unassigned Shard(s)" in this case. I don't mind some downtime on part of data (even on whole dataset) but i'd prefer not to reallocate data to other nodes and i'd not like to lose some of data permanently. What happen after the restarted node joins back to the cluster (Question 1.1) and i didn't use EXLUDE? Question 1.3: Same situation like in 1.1, but the node is restarted gracefully (by sending SIGTERM to service and start service again). Should I also send EXLUDE? What if there is not enough disc space on remaining nodes to hold data from a node that will restarted? Thank you in advance, Szymon ---
**Author:** ![Ben96](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ben96/32/28334_2.png) [@Ben96](https://discuss.elastic.co/u/Ben96)\ **Post date:** [March 5, 2018, 8:11am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-restarting-adding-node-s/122249/4 "2018-03-05T08:11:47Z")
Hi, 1.4 - Shards are in `Unassigned` State, after node restart and back to the cluster. They are automatically allocated to the right node after this one back online 1.3 - I don't think your node will get out of space, before your restart, elastic allocated space he need for your current (or old) Data. Not sure of my answer but i don't think a node could be in this state when restarting. That is a good question, i will take times to check it out. ---
**Author:** ![szymon](https://avatars.discourse-cdn.com/v4/letter/s/34f0e0/32.png) [@szymon](https://discuss.elastic.co/u/szymon)\ **Post date:** [March 6, 2018, 4:10pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-restarting-adding-node-s/122249/5 "2018-03-06T16:10:29Z")
Hello, once again, thanks Ben for quick response. I'm a little confused. Please let's go again through an example. "SHARD 0" is in "NODE A". We kill "NODE A", so "SHARD 0" is now in Unassigned State. **No data** was transfered/copied/reallocated to another box(es). "NODE A" is up again, so "SHARD 0" is now assigned to "NODE A". Still, no data was transfered/copied/reallocated to/from another box(es). Am i correct? So by > [@Ben96](#): > > They are automatically **allocated** to the right node after this one back online did you mean that shards are **assigned** to right node, but not transered/copied? And another example: "SHARD 0" is in "NODE A". We set "... exclude.\_name" : "NODE A", so "SHARD 0" **is** transfered/copied/reallocated to another box(es). But unfortunately, all other nodes all full. What happens with "SHARD 0"? Does it 'stay' in "NODE A"? If so, what happens after restart - same case as above? Thank you, Szymon ---
**Author:** ![Ben96](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ben96/32/28334_2.png) [@Ben96](https://discuss.elastic.co/u/Ben96)\ **Post date:** [March 6, 2018, 4:23pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-restarting-adding-node-s/122249/6 "2018-03-06T16:23:37Z")
Hello again 🙂 I just reproduced what your example try to show Explaining : 4 Nodes : `Node A-01 / Node A-02` `Node B-01 / Node A-02` Stopped "brutaly" Node A-01 with `systemctl stop elasticsearch-data-01.service` (my service name) Then `REPLICAS` on other`NODE` (like Node-A02) became `PRIMARY` Cluster is now `YELLOW` Got XX Unassigned shards (From Stopped Node) Then he "`relocate`" shards to other nodes (A & B) and he get back to `GREEN` When your Node back online he relocate shards again to use your full Data Node in the Cluster. ---
**Author:** ![szymon](https://avatars.discourse-cdn.com/v4/letter/s/34f0e0/32.png) [@szymon](https://discuss.elastic.co/u/szymon)\ **Post date:** [March 6, 2018, 4:47pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-restarting-adding-node-s/122249/7 "2018-03-06T16:47:39Z")
Thank you, but one more thing 🙂 Same situation, but we don't have replicas. How the cluster will behave? ---
**Author:** ![Ben96](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ben96/32/28334_2.png) [@Ben96](https://discuss.elastic.co/u/Ben96)\ **Post date:** [March 6, 2018, 4:53pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-restarting-adding-node-s/122249/8 "2018-03-06T16:53:04Z")
Got an idea but i'll test tomorrow 😉 ---
**Author:** ![Ben96](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ben96/32/28334_2.png) [@Ben96](https://discuss.elastic.co/u/Ben96)\ **Post date:** [March 7, 2018, 7:51am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-restarting-adding-node-s/122249/9 "2018-03-07T07:51:09Z")
Hi @szymon I did some test to reply your need. My Labs : Node-A01 - Only One Primary Shards & 0 Replicas for Indice Stopped Node-A01 ![Capture](https://us1.discourse-cdn.com/elastic/original/3X/4/3/43008160a4234b39f4d1b1bc2161d1d38acf5953.PNG) Cluster became RED No relocated shards incoming ---
**Author:** ![szymon](https://avatars.discourse-cdn.com/v4/letter/s/34f0e0/32.png) [@szymon](https://discuss.elastic.co/u/szymon)\ **Post date:** [March 7, 2018, 9:04am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-restarting-adding-node-s/122249/10 "2018-03-07T09:04:07Z")
Hi Ben, Great, thank you for responses and your time 🙂 ---
**Author:** ![Ben96](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ben96/32/28334_2.png) [@Ben96](https://discuss.elastic.co/u/Ben96)\ **Post date:** [March 7, 2018, 9:16am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-restarting-adding-node-s/122249/11 "2018-03-07T09:16:03Z")
No problem, here to help Elastic Comminuty 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 4, 2018, 9:16am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-restarting-adding-node-s/122249/12 "2018-04-04T09:16:34Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.