# Adding other nodes **URL:** \ **Category:** Elasticsearch\ **Created:** [July 17, 2019, 4:16pm UTC](https://discuss.elastic.co/t/adding-other-nodes/191057 "2019-07-17T16:16:10Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![rihad](https://avatars.discourse-cdn.com/v4/letter/r/a5b964/32.png) [@rihad](https://discuss.elastic.co/u/rihad)\ **Post date:** [July 17, 2019, 4:16pm UTC](https://discuss.elastic.co/t/adding-other-nodes/191057/1 "2019-07-17T16:16:10Z")
Hi, all. We want to add 2 more (master-eligible) nodes to our single-server cluster. Some experimentation on multiple dummy servers showed that the only thing I need to do for that is add discovery.zen.ping.unicast.hosts listing all three nodes. The problem is that we don't want to do it with any downtime, without having to restart the current master. Further experimentation showed that I can set the mentioned setting on a new server only, and it will happily join the cluster with the master that does NOT have that setting set. What does the fact that the current master doesn't have it set really mean in practical terms? Is it enough to set it in elasticsearch.yml for its restarts in the future and let it run without it for some time? Thanks. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 17, 2019, 7:28pm UTC](https://discuss.elastic.co/t/adding-other-nodes/191057/2 "2019-07-17T19:28:01Z")
> [@rihad](#): > > Is it enough to set it in elasticsearch.yml for its restarts in the future and let it run without it for some time? That's sufficient in version 7 (and later) but not in earlier versions. I think you're using an earlier version, because `discovery.zen.ping.unicast.hosts` is deprecated in v7 in favour of `discovery.seed_hosts`. In versions 6 and earlier you are right that you should set `discovery.zen.ping.unicast.hosts` on all three nodes, but the vitally important setting is [`discovery.zen.minimum_master_nodes`](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/discovery-settings.html#minimum_master_nodes). If you set that wrongly then your cluster can split into pieces and lose data as a result. You should set `discovery.zen.minimum_master_nodes: 2` in the two new nodes' config files. Then you should start **one** of the nodes and let it join the existing master. As soon as possible after that you should also set `discovery.zen.minimum_master_nodes` to `2` via the cluster settings API as well: ```plaintext PUT _cluster/settings {"persistent":{"discovery.zen.minimum_master_nodes": 2}} ``` Then, and only then, you should start the other new node and let it join the cluster. Once it's there, you should set `discovery.zen.minimum_master_nodes: 2` in the original master's config file and restart it. Once you've restarted this node you can safely remove the cluster setting: ```plaintext PUT _cluster/settings {"persistent":{"discovery.zen.minimum_master_nodes": null}} ``` ---
**Author:** ![rihad](https://avatars.discourse-cdn.com/v4/letter/r/a5b964/32.png) [@rihad](https://discuss.elastic.co/u/rihad)\ **Post date:** [July 18, 2019, 4:09am UTC](https://discuss.elastic.co/t/adding-other-nodes/191057/3 "2019-07-18T04:09:34Z")
Thanks a lot for the thorough reply. Yes, we're using 6.5.4, it's the only version available via FreeBSD ports. > [@DavidTurner](#): > > Once you've restarted this node you can safely remove the cluster setting: But shouldn't it be set to N/2+1 at all times? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 18, 2019, 5:54am UTC](https://discuss.elastic.co/t/adding-other-nodes/191057/4 "2019-07-18T05:54:49Z")
> [@rihad](#): > > But shouldn't it be set to N/2+1 at all times? Yes, but by this point in the process it is set to 2 on each node, so there's no need for it also to be set in the cluster settings. ---
**Author:** ![rihad](https://avatars.discourse-cdn.com/v4/letter/r/a5b964/32.png) [@rihad](https://discuss.elastic.co/u/rihad)\ **Post date:** [July 18, 2019, 6:11am UTC](https://discuss.elastic.co/t/adding-other-nodes/191057/5 "2019-07-18T06:11:31Z")
One last question if I may: can we set `discovery.zen.ping.unicast.hosts` dynamically on the current master without having to restart it? And simply repeat the setting in it elasticsearch.yml for it to persist. It's just that our current Ruby on Rails client code isn't ready to cycle through all possible ES servers and only uses the single master node. We're planning on extending that a bit later. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 18, 2019, 6:15am UTC](https://discuss.elastic.co/t/adding-other-nodes/191057/6 "2019-07-18T06:15:57Z")
No, that's not possible. This setting can only be set in `elasticsearch.yml`. ---
**Author:** ![rihad](https://avatars.discourse-cdn.com/v4/letter/r/a5b964/32.png) [@rihad](https://discuss.elastic.co/u/rihad)\ **Post date:** [July 18, 2019, 6:53am UTC](https://discuss.elastic.co/t/adding-other-nodes/191057/7 "2019-07-18T06:53:46Z")
Since setting unicast.hosts on the current master and restarting it should be done quickly after bringing in the other 2 nodes, can I do them backwards? Take the 2 new nodes down, set unicast.hosts on the current master first, restart it, see if it still works, and then, maybe a day later, bring the first server up with unicast.hosts properly configured and minimum\_master\_nodes set to 2, also set minimum\_master\_nodes to 2 via the API for the master to be aware of it, repeating it in its elasticsearch.yml, and finally after their status turns green run the 3rd node with unicast.hosts minimum\_master\_nodes set same as on the 2nd node? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 18, 2019, 7:24am UTC](https://discuss.elastic.co/t/adding-other-nodes/191057/8 "2019-07-18T07:24:44Z")
I think you are perhaps confusing `discovery.zen.ping.unicast.hosts` and `discovery.zen.minimum_master_nodes`? It's very important to set `minimum_master_nodes` correctly as soon as possible, but the `unicast.hosts` setting is quite tolerant of being set wrongly. It particularly tolerates having extra entries that don't (yet) correspond with running nodes. ---
**Author:** ![rihad](https://avatars.discourse-cdn.com/v4/letter/r/a5b964/32.png) [@rihad](https://discuss.elastic.co/u/rihad)\ **Post date:** [July 18, 2019, 7:37am UTC](https://discuss.elastic.co/t/adding-other-nodes/191057/9 "2019-07-18T07:37:32Z")
Sorry, then I probably misunderstood you when you wrote that I absolutely had to restart the current master. The only reason to restart it would be to apply the unicast.hosts change on it, because minimum\_master\_nodes can be set dymamically also. But now I see that unicast.hosts will probably be needed if a network outage separates current master from the other two: they would elect a new master between them (since their number suffices minimum\_master\_nodes=2), and the current master would remain the master not knowing of other parties because of the unchanged unicast.hosts - split brain. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 18, 2019, 8:17am UTC](https://discuss.elastic.co/t/adding-other-nodes/191057/10 "2019-07-18T08:17:03Z")
> [@rihad](#): > > the current master would remain the master not knowing of other parties because of the unchanged unicast.hosts Yes, this still indicates some confusion over `unicast.hosts` vs `minimum_master_nodes`. The reason a master might remain a master on its own is **because it thinks `minimum_master_nodes` is 1** and nothing at all to do with `unicast.hosts`. You could put 3, or 10, or 100 entries in `unicast.hosts`, and it would make no difference to this situation. ---
**Author:** ![rihad](https://avatars.discourse-cdn.com/v4/letter/r/a5b964/32.png) [@rihad](https://discuss.elastic.co/u/rihad)\ **Post date:** [July 18, 2019, 8:20am UTC](https://discuss.elastic.co/t/adding-other-nodes/191057/11 "2019-07-18T08:20:11Z")
Can I set minimum\_master\_nodes=2 on the master before doing anything else? It wouldn't prevent it from functioning in a single-server scenario? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 18, 2019, 8:23am UTC](https://discuss.elastic.co/t/adding-other-nodes/191057/12 "2019-07-18T08:23:31Z")
No - that's the problem. If you were to set `minimum_master_nodes: 2` on a solitary master then it wouldn't be able to be master any more, because with that setting you require at least two master-eligible nodes. ---
**Author:** ![rihad](https://avatars.discourse-cdn.com/v4/letter/r/a5b964/32.png) [@rihad](https://discuss.elastic.co/u/rihad)\ **Post date:** [July 19, 2019, 7:33am UTC](https://discuss.elastic.co/t/adding-other-nodes/191057/13 "2019-07-19T07:33:43Z")
Wow, does that mean that if in the future 2 of the 3 master-eligible servers die off or become unreachable, the single remaning server will be unable to function if it happens NOT to be the master? What's the point in having fault tolerance then? ) ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 19, 2019, 8:09am UTC](https://discuss.elastic.co/t/adding-other-nodes/191057/14 "2019-07-19T08:09:25Z")
Not quite. If 2 of the 3 master-eligible nodes fail then the remaining node will not function as a master _whether it was the master beforehand or not_. It'd have no way of knowing if the 2 other nodes are really gone or whether it's just suffering from a network partition, and therefore it cannot tell whether the other 2 nodes have formed themselves into a cluster on the other side of the partition. The only way to avoid a split-brain situation (i.e. data loss) is to stand down. If you have 3 master-eligible nodes then you can tolerate the loss of at most one of them. This isn't a limitation imposed by Elasticsearch - it's theoretically optimal. If you want to be able to tolerate the loss of 2 masters then you need at least 5 master-eligible nodes. ---
**Author:** ![rihad](https://avatars.discourse-cdn.com/v4/letter/r/a5b964/32.png) [@rihad](https://discuss.elastic.co/u/rihad)\ **Post date:** [July 24, 2019, 11:18am UTC](https://discuss.elastic.co/t/adding-other-nodes/191057/15 "2019-07-24T11:18:30Z")
I let them join the cluster and cluster status turned green. I'm a bit embarrassed by the fact that disk usage (du -sh) of /var/db/elasticsearch is 17G on the master and only 5.8-5.9GB on the slaves. $ curl -X GET [http://titan.local:9200/\_cluster/health](http://titan.local:9200/_cluster/health) {"cluster\_name":"foo","status":"green","timed\_out":false,"number\_of\_nodes":3,"number\_of\_data\_nodes":3,"active\_primary\_shards":5,"active\_shards":10,"relocating\_shards":0,"initializing\_shards":0,"unassigned\_shards":0,"delayed\_unassigned\_shards":0,"number\_of\_pending\_tasks":0,"number\_of\_in\_flight\_fetch":0,"task\_max\_waiting\_in\_queue\_millis":0,"active\_shards\_percent\_as\_number":100.0} ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 21, 2019, 11:18am UTC](https://discuss.elastic.co/t/adding-other-nodes/191057/16 "2019-08-21T11:18:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.