# Difficulty adding nodes **URL:** \ **Category:** Elasticsearch\ **Created:** [June 12, 2019, 12:11am UTC](https://discuss.elastic.co/t/difficulty-adding-nodes/185303 "2019-06-12T00:11:17Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Brett\_Blackham](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brett_blackham/32/47906_2.png) [@Brett\_Blackham](https://discuss.elastic.co/u/Brett_Blackham)\ **Post date:** [June 12, 2019, 12:11am UTC](https://discuss.elastic.co/t/difficulty-adding-nodes/185303/1 "2019-06-12T00:11:17Z")
I've been following several tutorials/examples but I am not able to get elastic to see other nodes. I have 3 VMs running Ubuntu 18.04 LTS /etc/hosts 10.192.10.61 elastic1 10.192.10.62 elastic2 10.192.10.63 elastic3 I can telnet to port 9200 and 9300 on all three servers. But I can never get it to show more than one node. I imagine I'm missing something silly in my config. ``` root@elastic1:/var/log# curl -XGET 'http://10.192.10.61:9200/_cluster/health?pretty' { "cluster_name" : "the-cluster", "status" : "red", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "active_primary_shards" : 0, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : "NaN" } ``` Here are the config files ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/d/7dbd03a4df75ffd76accb742543838af09940f37.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/8/f805e679917b86555971744c561c070f6236e9f2.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/a/0a4f527b4ef70a92affbac047f5b393b189c698c.png) Tail of the cluster log [2019-06-12T00:03:06,983][INFO][o.e.p.PluginsService] [elastic1] loaded module [x-pack-security] [2019-06-12T00:03:06,984][INFO][o.e.p.PluginsService] [elastic1] loaded module [x-pack-sql] [2019-06-12T00:03:06,984][INFO][o.e.p.PluginsService] [elastic1] loaded module [x-pack-watcher] [2019-06-12T00:03:06,985][INFO][o.e.p.PluginsService] [elastic1] no plugins loaded [2019-06-12T00:03:11,566][INFO][o.e.x.m.p.l.CppLogMessageHandler] [elastic1] [controller/5736] [Main.cc@109] controller (64 bit): Version 7.1.1 (Build fd619a36eb77df) Copyright (c) 2019 Elasticsearch BV [2019-06-12T00:03:12,369][INFO][o.e.d.DiscoveryModule] [elastic1] using discovery type [zen] and seed hosts providers [settings] [2019-06-12T00:03:13,324][INFO][o.e.n.Node] [elastic1] initialized [2019-06-12T00:03:13,325][INFO][o.e.n.Node] [elastic1] starting ... [2019-06-12T00:03:13,490][INFO][o.e.t.TransportService] [elastic1] publish\_address {10.192.10.61:9300}, bound\_addresses {10.192.10.61:9300} [2019-06-12T00:03:13,499][INFO][o.e.b.BootstrapChecks] [elastic1] bound or publishing to a non-loopback address, enforcing bootstrap checks [2019-06-12T00:03:13,507][INFO][o.e.c.c.Coordinator] [elastic1] cluster UUID [8V5kNevdQDuGgqMCCjRg5A] [2019-06-12T00:03:13,667][INFO][o.e.c.s.MasterService] [elastic1] elected-as-master ([1] nodes joined)[{elastic1}{DpJg-1UISCSBF\_YB8l9Ywg}{KXS9YGBKRDiN1sNzYNOtMQ}{10.192.10.61}{10.192.10.61:9300}{ml.machine\_memory=16819339264, xpack.installed=true, ml.max\_ope n\_jobs=20} elect leader, _BECOME\_MASTER\_TASK_, _FINISH\_ELECTION_], term: 37, version: 95, reason: master node changed {previous , current [{elastic1}{DpJg-1UISCSBF\_YB8l9Ywg}{KXS9YGBKRDiN1sNzYNOtMQ}{10.192.10.61}{10.192.10.61:9300}{ml.machine\_memory=16819339264, xpack.installed=true, ml.max\_open\_jobs=20}]} [2019-06-12T00:03:13,729][INFO][o.e.c.s.ClusterApplierService] [elastic1] master node changed {previous , current [{elastic1}{DpJg-1UISCSBF\_YB8l9Ywg}{KXS9YGBKRDiN1sNzYNOtMQ}{10.192.10.61}{10.192.10.61:9300}{ml.machine\_memory=16819339264, xpack.installed=true, ml.max\_open\_jobs=20}]}, term: 37, version: 95, reason: Publication{term=37, version=95} [2019-06-12T00:03:13,806][INFO][o.e.h.AbstractHttpServerTransport] [elastic1] publish\_address {10.192.10.61:9200}, bound\_addresses {10.192.10.61:9200} [2019-06-12T00:03:13,806][INFO][o.e.n.Node] [elastic1] started [2019-06-12T00:03:45,840][INFO][o.e.m.j.JvmGcMonitorService] [elastic1] [gc][young][32][3] duration [778ms], collections [1]/[1.4s], total [778ms]/[844ms], memory [582.4mb]-\>[212.8mb]/[1.9gb], all\_pools {[young] [527.1mb]-\>[19.5mb]/[532.5mb]}{[survivor] [55.3mb] -\>[56.1mb]/[66.5mb]}{[old] [0b]-\>[138.9mb]/[1.3gb]} [2019-06-12T00:03:45,841][WARN][o.e.m.j.JvmGcMonitorService] [elastic1] [gc][32] overhead, spent [778ms] collecting in the last [1.4s] Can anyone see what I'm missing? ---
**Author:** ![shin-higuchi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shin-higuchi/32/46524_2.png) [@shin-higuchi](https://discuss.elastic.co/u/shin-higuchi)\ **Post date:** [June 12, 2019, 3:26am UTC](https://discuss.elastic.co/t/difficulty-adding-nodes/185303/2 "2019-06-12T03:26:02Z")
Hi, I guess you need to configure "cluster.initial\_master\_nodes" in your elasticsearch.yml. Ref: [https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-bootstrap-cluster.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-bootstrap-cluster.html) Thanks. ---
**Author:** ![Brett\_Blackham](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brett_blackham/32/47906_2.png) [@Brett\_Blackham](https://discuss.elastic.co/u/Brett_Blackham)\ **Post date:** [June 12, 2019, 1:23pm UTC](https://discuss.elastic.co/t/difficulty-adding-nodes/185303/3 "2019-06-12T13:23:07Z")
Thanks for looking. I have tried it with cluster.initial\_master\_nodes commented out. Also with just elastict1, and with elastic2 and elastic3. Same results every time. The only setting that seems to change the \_cluster/health is gateway.recover\_after\_nodes. Setting it to 2 will start but with a "status": "red". A value of 1 (or commented out) will start with a "status": "green". I found someone else that was getting this same issue. In fact, I recognize his configs. He used one of the 5 different example/tutorials I saw. So, I'm not the only one running into this. [He had the same, or similar, issue](https://discuss.elastic.co/t/elasticsearch-7-cluster-setup-with-3-nodes/177016/5) Hari.v said he got his cluster to work by adding "node.master: true , node.data: true for all the 3 nodes". I modified my config to mach his (again), but now I have three ES servers each running as individual (single node) clusters. Below is the log file from elastic2, started after elastic1. Elastic1 has the exact same log message, but with it's own IP: ``` [2019-06-12T13:13:47,739][INFO][o.e.n.Node] [elastic2] starting ... [2019-06-12T13:13:47,887][INFO][o.e.t.TransportService] [elastic2] publish_address {10.192.10.62:9300}, bound_addresses {10.192.10.62:9300} [2019-06-12T13:13:47,895][INFO][o.e.b.BootstrapChecks] [elastic2] bound or publishing to a non-loopback address, enforcing bootstrap checks [2019-06-12T13:13:47,903][INFO][o.e.c.c.Coordinator] [elastic2] cluster UUID [gnWCd64oQlenU0d1eVaL6Q] [2019-06-12T13:13:48,012][INFO][o.e.c.s.MasterService] [elastic2] elected-as-master ([1] nodes joined)[{elastic2}{ML6K63J2Tvmx-WsPqeaeHA}{32880EVrSFyvCKy2evsXyA}{10.192.10.62}{10.192.10.62:9300}{ml.machine_memory=1 6819339264, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 45, version: 119, reason: master node changed {previous [], current [{elastic2}{ML6K63J2Tvmx-WsPqeaeHA }{32880EVrSFyvCKy2evsXyA}{10.192.10.62}{10.192.10.62:9300}{ml.machine_memory=16819339264, xpack.installed=true, ml.max_open_jobs=20}]} [2019-06-12T13:13:48,072][INFO][o.e.c.s.ClusterApplierService] [elastic2] master node changed {previous [], current [{elastic2}{ML6K63J2Tvmx-WsPqeaeHA}{32880EVrSFyvCKy2evsXyA}{10.192.10.62}{10.192.10.62:9300}{ml.machin e_memory=16819339264, xpack.installed=true, ml.max_open_jobs=20}]}, term: 45, version: 119, reason: Publication{term=45, version=119} [2019-06-12T13:13:48,176][INFO][o.e.h.AbstractHttpServerTransport] [elastic2] publish_address {10.192.10.62:9200}, bound_addresses {10.192.10.62:9200} [2019-06-12T13:13:48,176][INFO][o.e.n.Node] [elastic2] started [2019-06-12T13:13:48,401][INFO][o.e.l.LicenseService] [elastic2] license [accb73b4-6b25-43ec-a3e4-a73575d5e69c] mode [basic] - valid [2019-06-12T13:13:48,413][INFO][o.e.g.GatewayService] [elastic2] recovered [0] indices into cluster_state ``` ---
**Author:** ![Brett\_Blackham](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brett_blackham/32/47906_2.png) [@Brett\_Blackham](https://discuss.elastic.co/u/Brett_Blackham)\ **Post date:** [June 12, 2019, 1:26pm UTC](https://discuss.elastic.co/t/difficulty-adding-nodes/185303/4 "2019-06-12T13:26:13Z")
Here are the current configs. ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/6/f6ef030bf8c40f4a34621932533a2354eba2312c.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/2/e2c4cc7c7695966deb88415a14dd427c11009c62.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/9/790a37c8616a24784c9da49bd2c57b483b489089.png) ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 12, 2019, 1:32pm UTC](https://discuss.elastic.co/t/difficulty-adding-nodes/185303/5 "2019-06-12T13:32:20Z")
I see different cluster UUIDs in two of the logs you shared, indicating you have formed different clusters. I think [this note in the manual](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-bootstrap-cluster.html#modules-discovery-bootstrap-cluster-joining) applies. ---
**Author:** ![Brett\_Blackham](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brett_blackham/32/47906_2.png) [@Brett\_Blackham](https://discuss.elastic.co/u/Brett_Blackham)\ **Post date:** [June 12, 2019, 1:44pm UTC](https://discuss.elastic.co/t/difficulty-adding-nodes/185303/6 "2019-06-12T13:44:46Z")
Mr Turner. I'm so happy I could kiss you. But we both might like that, which could be strange. So I'll just post my steps to get it working. On all three servers I ran the following as root: > root@elastic1:~# sudo systemctl stop elasticsearch.service > root@elastic1:/etc/elasticsearch# cd /var/lib/elasticsearch/ > root@elastic1:/var/lib/elasticsearch# ls > nodes > root@elastic1:/var/lib/elasticsearch# rm -r nodes Then I "systemclt start elasticserach.service" on all three, but not before I had deleted the contents of path.data. ``` { "cluster_name" : "the-cluster", "status" : "green", "timed_out" : false, "number_of_nodes" : 3, "number_of_data_nodes" : 3, "active_primary_shards" : 0, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } ``` Thank you again. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 12, 2019, 9:50pm UTC](https://discuss.elastic.co/t/difficulty-adding-nodes/185303/7 "2019-06-12T21:50:29Z")
For future reference, please don't post pictures of text, they are difficult to read, impossible to search and some people may not be even able to see them 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 10, 2019, 9:50pm UTC](https://discuss.elastic.co/t/difficulty-adding-nodes/185303/8 "2019-07-10T21:50:38Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.