# \[Elasticsearch 7.1.1\] Node failed to join cluster with GCE discovery plugin due to different cluster uuid **URL:** \ **Category:** Elasticsearch\ **Created:** [July 19, 2019, 1:10am UTC](https://discuss.elastic.co/t/elasticsearch-7-1-1-node-failed-to-join-cluster-with-gce-discovery-plugin-due-to-different-cluster-uuid/191294 "2019-07-19T01:10:29Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![haotian9850](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@haotian9850](https://discuss.elastic.co/u/haotian9850)\ **Post date:** [July 19, 2019, 1:10am UTC](https://discuss.elastic.co/t/elasticsearch-7-1-1-node-failed-to-join-cluster-with-gce-discovery-plugin-due-to-different-cluster-uuid/191294/1 "2019-07-19T01:10:29Z")
Hey there, I am currently making an automated elasticsearch cluster deployment on Google Compute Platform with terraform and packer. For a testing 2-node test deployment, the config files (`elasticsearch.yml`) were as below: ` node-1 (private IP = 10.15.0.80) path.logs: /elasticsearch/logs cluster.name: testing-es-cluster-gce discovery.zen.minimum_master_nodes: 1 cluster.initial_master_nodes: [node-10.15.0.80] cloud.gce.project_id: dev-cortex-global-01 discovery.seed_providers: gce http.port: 9200 node.master: true path.data: /elasticsearch/data network.host: 10.15.0.80 node.name: node-10.15.0.80 cloud.gce.zone: us-central1-a ` and this is node 2's `elasticsearch.yml` ` path.logs: /elasticsearch/logs cluster.name: testing-es-cluster-gce discovery.zen.minimum_master_nodes: 1 cluster.initial_master_nodes:[node-10.15.0.95] cloud.gce.project_id: dev-cortex-global-01 discovery.seed_providers: gce http.port: 9200 node.master: true path.data: /elasticsearch/data network.host: 10.15.0.95 node.name: node-10.15.0.95 cloud.gce.zone: us-central1-a ` The above 2 nodes (VM instances) are deployed in the same GCP project and zone and both of their port `9200` and `9300` are not blocked by anything. However, it seems when I start elasticsearch (which I configured as a systemd service) by `systemctl start elasticsearch` on both instances neither is able to discover the other. The log from node 1 is as follow ` ... Jul 19 01:03:29 haotian-tf-es-node-0 elasticsearch[2766]: [2019-07-19T01:03:29,399][INFO][o.e.c.g.GceInstancesServiceImpl] [node-10.15.0.95] starting GCE discovery service Jul 19 01:03:29 haotian-tf-es-node-0 elasticsearch[2766]: [2019-07-19T01:03:29,648][INFO][o.e.c.s.MasterService] [node-10.15.0.95] elected-as-master ([1] nodes joined)[{node-10.15.0.95}{uu1BhjLbSQOS3JLUbQprQQ}{cUeLygwyRK688mApErNa1Q}{10.15.0 .95}{10.15.0.95:9300}{ml.machine_memory=7835996160, xpack.installed=true, ml.max_open_jobs=20} elect leader, BECOME_MASTER_TASK, FINISH_ELECTION], term: 1, version: 1, reason: master node changed {previous , current [{node-10.15.0.95}{uu1Bh jLbSQOS3JLUbQprQQ}{cUeLygwyRK688mApErNa1Q}{10.15.0.95}{10.15.0.95:9300}{ml.machine_memory=7835996160, xpack.installed=true, ml.max_open_jobs=20}]} Jul 19 01:03:29 haotian-tf-es-node-0 elasticsearch[2766]: [2019-07-19T01:03:29,744][INFO][o.e.c.c.CoordinationState] [node-10.15.0.95] cluster UUID set to [7BBzJqO4QDKsCpYDpfJ1RQ] Jul 19 01:03:29 haotian-tf-es-node-0 elasticsearch[2766]: [2019-07-19T01:03:29,793][INFO][o.e.c.s.ClusterApplierService] [node-10.15.0.95] master node changed {previous , current [{node-10.15.0.95}{uu1BhjLbSQOS3JLUbQprQQ}{cUeLygwyRK688mApErNa1 Q}{10.15.0.95}{10.15.0.95:9300}{ml.machine_memory=7835996160, xpack.installed=true, ml.max_open_jobs=20}]}, term: 1, version: 1, reason: Publication{term=1, version=1} Jul 19 01:03:30 haotian-tf-es-node-0 elasticsearch[2766]: [2019-07-19T01:03:30,039][WARN][o.e.x.s.a.s.m.NativeRoleMappingStore] [node-10.15.0.95] Failed to clear cache for realms [] Jul 19 01:03:30 haotian-tf-es-node-0 elasticsearch[2766]: [2019-07-19T01:03:30,049][INFO][o.e.h.AbstractHttpServerTransport] [node-10.15.0.95] publish_address {10.15.0.95:9200}, bound_addresses {10.15.0.95:9200} Jul 19 01:03:30 haotian-tf-es-node-0 elasticsearch[2766]: [2019-07-19T01:03:30,051][INFO][o.e.n.Node] [node-10.15.0.95] started Jul 19 01:03:30 haotian-tf-es-node-0 elasticsearch[2766]: [2019-07-19T01:03:30,428][INFO][o.e.g.GatewayService] [node-10.15.0.95] recovered [0] indices into cluster_state ` and node 2's log: ` ... Jul 19 01:03:43 haotian-tf-es-node-1 elasticsearch[2998]: [2019-07-19T01:03:43,813][INFO][o.e.c.g.GceInstancesServiceImpl] [node-10.15.0.80] starting GCE discovery service Jul 19 01:03:44 haotian-tf-es-node-1 elasticsearch[2998]: [2019-07-19T01:03:44,155][INFO][o.e.c.s.MasterService] [node-10.15.0.80] elected-as-master ([1] nodes joined)[{node-10.15.0.80}{xRijgHF8SXWh36jZ9OVmZg}{YeUdSQ5RQWybnN-dPRD4Xg}{10.15.0 .80}{10.15.0.80:9300}{ml.machine_memory=7836004352, xpack.installed=true, ml.max_open_jobs=20} elect leader, BECOME_MASTER_TASK, FINISH_ELECTION], term: 1, version: 1, reason: master node changed {previous , current [{node-10.15.0.80}{xRijg HF8SXWh36jZ9OVmZg}{YeUdSQ5RQWybnN-dPRD4Xg}{10.15.0.80}{10.15.0.80:9300}{ml.machine_memory=7836004352, xpack.installed=true, ml.max_open_jobs=20}]} Jul 19 01:03:44 haotian-tf-es-node-1 elasticsearch[2998]: [2019-07-19T01:03:44,234][INFO][o.e.c.c.CoordinationState] [node-10.15.0.80] cluster UUID set to [F2q29Cc9So-3Wt-hJm8R1w] Jul 19 01:03:44 haotian-tf-es-node-1 elasticsearch[2998]: [2019-07-19T01:03:44,260][INFO][o.e.c.s.ClusterApplierService] [node-10.15.0.80] master node changed {previous , current [{node-10.15.0.80}{xRijgHF8SXWh36jZ9OVmZg}{YeUdSQ5RQWybnN-dPRD4X g}{10.15.0.80}{10.15.0.80:9300}{ml.machine_memory=7836004352, xpack.installed=true, ml.max_open_jobs=20}]}, term: 1, version: 1, reason: Publication{term=1, version=1} Jul 19 01:03:44 haotian-tf-es-node-1 elasticsearch[2998]: [2019-07-19T01:03:44,449][INFO][o.e.h.AbstractHttpServerTransport] [node-10.15.0.80] publish_address {10.15.0.80:9200}, bound_addresses {10.15.0.80:9200} Jul 19 01:03:44 haotian-tf-es-node-1 elasticsearch[2998]: [2019-07-19T01:03:44,451][INFO][o.e.n.Node] [node-10.15.0.80] started Jul 19 01:03:44 haotian-tf-es-node-1 elasticsearch[2998]: [2019-07-19T01:03:44,454][WARN][o.e.x.s.a.s.m.NativeRoleMappingStore] [node-10.15.0.80] Failed to clear cache for realms [] Jul 19 01:03:44 haotian-tf-es-node-1 elasticsearch[2998]: [2019-07-19T01:03:44,660][INFO][o.e.g.GatewayService] [node-10.15.0.80] recovered [0] indices into cluster_state ` After a little research I discovered that it might be the different cluster uuid that is preventing node from joining the cluster. Yet the cluster name of both nodes are set to the same (via another systemd service that has a `Before=` dependency on the elasticsearch service - so `elasticsearch.yml` will always be written before elasticsearch is started on each node). I am wondering what could be a possible cause of the inconsistency over cluster uuid (that I suspect is causing node not being able to join)? What can I do to fix it? Thanks!!! ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 19, 2019, 6:56am UTC](https://discuss.elastic.co/t/elasticsearch-7-1-1-node-failed-to-join-cluster-with-gce-discovery-plugin-due-to-different-cluster-uuid/191294/2 "2019-07-19T06:56:51Z")
You have instructed the nodes to form separate one-node clusters: > [@haotian9850](#): > > ```auto > cluster.initial_master_nodes: [node-10.15.0.80] > > ``` > > vs > > ```auto > cluster.initial_master_nodes:[node-10.15.0.95] > > ``` From [the docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-bootstrap-cluster.html): > **WARNING** : You must set `cluster.initial_master_nodes` to the same list of nodes on each node on which it is set in order to be sure that only a single cluster forms during bootstrapping and therefore to avoid the risk of data loss. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 19, 2019, 7:12am UTC](https://discuss.elastic.co/t/elasticsearch-7-1-1-node-failed-to-join-cluster-with-gce-discovery-plugin-due-to-different-cluster-uuid/191294/3 "2019-07-19T07:12:11Z")
> [@haotian9850](#): > > `discovery.zen.minimum_master_nodes: 1` Also this setting is deprecated and does nothing in a new v7 cluster. If this were a v6-or-earlier cluster then setting it to `1` like this would lead to data loss. I suggest you remove this line entirely. ---
**Author:** ![haotian9850](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@haotian9850](https://discuss.elastic.co/u/haotian9850)\ **Post date:** [July 19, 2019, 7:55am UTC](https://discuss.elastic.co/t/elasticsearch-7-1-1-node-failed-to-join-cluster-with-gce-discovery-plugin-due-to-different-cluster-uuid/191294/4 "2019-07-19T07:55:02Z")
Hi David, thank you so much for your reply! I think your suggestion would solve the issue. However, as I am doing fully automated builds deployed on GCP (there is no way that I can assign each node a unique name as they are coming from the same packer machine image before booting them up - I have another provisioning script that writes to `elasticsearch.yml` and name the node after its private IP address when the instances get started first time), I am wondering if there is a workaround that does not require specifying `cluster.initial_master_modes` in `elasticsearch.yml` (similarly, `discovery.seed_hosts` wouldnt work since the node IPs are not known to me beforehand)? Thanks! ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 19, 2019, 8:17am UTC](https://discuss.elastic.co/t/elasticsearch-7-1-1-node-failed-to-join-cluster-with-gce-discovery-plugin-due-to-different-cluster-uuid/191294/5 "2019-07-19T08:17:43Z")
No, `cluster.initial_master_nodes` is required on at least one node, and must be set consistently on all nodes on which it is set. I don't think that GCP offers an API that could be used for this, so the `gce-discovery` plugin cannot really help to auto-discover a suitable value. Note that it's only required when provisioning a brand-new cluster and should be removed from the config once the cluster is up and running. I think the simplest solution is to fix the names of the first few master-eligible nodes up front rather than generating them dynamically like you are doing. ---
**Author:** ![haotian9850](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@haotian9850](https://discuss.elastic.co/u/haotian9850)\ **Post date:** [July 19, 2019, 7:40pm UTC](https://discuss.elastic.co/t/elasticsearch-7-1-1-node-failed-to-join-cluster-with-gce-discovery-plugin-due-to-different-cluster-uuid/191294/6 "2019-07-19T19:40:05Z")
Thanks David! I think I will just implement another service that can write the full `cluster.initial_master_nodes` to `elasticsearch.yml`. Can a multi-node cluster be bootstrapped when only **one** mater-eligible node has `cluster.initial_master_nodes` set (the others will not have this entry in their `.yml` config)? I am trying to minimize the number of http requests I have to make as for a large cluster writing to every node might potentially consume a large amount of time. Thanks! ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 19, 2019, 11:26pm UTC](https://discuss.elastic.co/t/elasticsearch-7-1-1-node-failed-to-join-cluster-with-gce-discovery-plugin-due-to-different-cluster-uuid/191294/7 "2019-07-19T23:26:27Z")
> [@haotian9850](#): > > Can a multi-node cluster be bootstrapped when only **one** mater-eligible node has `cluster.initial_master_nodes` set (the others will not have this entry in their `.yml` config)? Yes, although if that node fails before the cluster has fully formed then you may have to start again from scratch. However ... > [@haotian9850](#): > > I am trying to minimize the number of http requests I have to make as for a large cluster writing to every node might potentially consume a large amount of time. ... this setting should only be set on master-eligible nodes and you should not have very many of them. It has no effect on nodes with `node.master: false`. ---
**Author:** ![haotian9850](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@haotian9850](https://discuss.elastic.co/u/haotian9850)\ **Post date:** [July 25, 2019, 6:18pm UTC](https://discuss.elastic.co/t/elasticsearch-7-1-1-node-failed-to-join-cluster-with-gce-discovery-plugin-due-to-different-cluster-uuid/191294/8 "2019-07-25T18:18:20Z")
Thanks David, the problem is now resolved! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 22, 2019, 6:18pm UTC](https://discuss.elastic.co/t/elasticsearch-7-1-1-node-failed-to-join-cluster-with-gce-discovery-plugin-due-to-different-cluster-uuid/191294/9 "2019-08-22T18:18:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.