# Nodes not communicating in 3-node cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [June 12, 2019, 10:56am UTC](https://discuss.elastic.co/t/nodes-not-communicating-in-3-node-cluster/185393 "2019-06-12T10:56:15Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Paul\_14](https://avatars.discourse-cdn.com/v4/letter/p/8edcca/32.png) [@Paul\_14](https://discuss.elastic.co/u/Paul_14)\ **Post date:** [June 12, 2019, 10:56am UTC](https://discuss.elastic.co/t/nodes-not-communicating-in-3-node-cluster/185393/1 "2019-06-12T10:56:15Z")
Hello, I have deployed a 3 node Elasticsearch 7.0 Cluster from the Azure marketplace. However, I'm having trouble configuring them to see each other. Below I have detailed the elasticsearch.yml for 2 of the nodes. Node 1 is called esp-data-0 (192.248.16.7). Here is the elasticsearch.yml > ``` > cluster.name: "elasticsearch" > node.name: "esp-data-0" > path.logs: /var/log/elasticsearch > path.data: /datadisks/disk1/elasticsearch/data > cluster.initial_master_nodes: ["esp-data-0","esp-data-1","esp-data-2"] > discovery.seed_hosts: ["192.248.16.7","192.248.16.6","192.248.16.8"] > network.bind_host: "192.248.16.7" > network.publish_host: "192.248.16.7" > transport.port: "9300" > node.master: true > node.data: true > discovery.zen.minimum_master_nodes: 2 > network.host: ["192.248.16.7",_local_] > node.max_local_storage_nodes: 1 > node.attr.fault_domain: 1 > node.attr.update_domain: 1 > cluster.routing.allocation.awareness.attributes: fault_domain,update_domain > xpack.license.self_generated.type: trial > xpack.security.enabled: false > bootstrap.memory_lock: true > > ``` Nodes 2 and 3 also have master and data set to true. See node 2 below: > ``` > cluster.name: "elasticsearch" > node.name: "esp-data-1" > path.logs: /var/log/elasticsearch > path.data: /datadisks/disk1/elasticsearch/data > cluster.initial_master_nodes: ["esp-data-0","esp-data-1","esp-data-2"] > discovery.seed_hosts: ["192.248.16.7","192.248.16.6","192.248.16.8"] > network.bind_host: "192.248.16.6" > network.publish_host: "192.248.16.6" > transport.port: "9300" > node.master: true > node.data: true > discovery.zen.minimum_master_nodes: 2 > network.host: ["192.248.16.6",_local_] > node.max_local_storage_nodes: 1 > node.attr.fault_domain: 0 > node.attr.update_domain: 0 > cluster.routing.allocation.awareness.attributes: fault_domain,update_domain > xpack.license.self_generated.type: trial > xpack.security.enabled: false > bootstrap.memory_lock: true > > ``` On all three nodes, when I run the below to view the cluster topology, they only appear to have sight of themselves and not the other 2 nodes in the cluster... I CAN however, telnet successfully from each node to the other on ports 9200/9300 so I believe the issue must reside in my Elasticsearch congig. > ``` > telnet 192.248.16.6 9200 > Trying 192.248.16.6... > Connected to 192.248.16.6. > > ``` > ``` > telnet 192.248.16.6 9300 > Trying 192.248.16.6... > Connected to 192.248.16.6. > > ``` Can you please suggest why this may be? Result of running nodes command on node 1: curl 192.248.16.7:9200/\_cat/nodes?v ``` ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name 192.248.16.7 3 68 0 0.00 0.02 0.00 mdi * esp-data-0 ``` Result of running nodes command on node 2: curl 192.248.16.6:9200/\_cat/nodes?v ``` ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name 192.248.16.6 3 68 0 0.00 0.02 0.00 mdi * esp-data-1 ``` ---
**Author:** ![CristianoFerreira](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cristianoferreira/32/47529_2.png) [@CristianoFerreira](https://discuss.elastic.co/u/CristianoFerreira)\ **Post date:** [June 12, 2019, 12:06pm UTC](https://discuss.elastic.co/t/nodes-not-communicating-in-3-node-cluster/185393/2 "2019-06-12T12:06:30Z")
Hi, Try change `cluster.initial_master_nodes: ["esp-data-0","esp-data-1","esp-data-2"]` to `cluster.initial_master_nodes: ["192.248.16.7","192.248.16.6","192.248.16.8"]`. Please provide the Elasticsearch log output to better understand the problem. Best Regards, ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 12, 2019, 12:11pm UTC](https://discuss.elastic.co/t/nodes-not-communicating-in-3-node-cluster/185393/3 "2019-06-12T12:11:07Z")
> [@CristianoFerreira](#): > > Try change `cluster.initial_master_nodes: ["esp-data-0","esp-data-1","esp-data-2"]` to `cluster.initial_master_nodes: ["192.248.16.7","192.248.16.6","192.248.16.8"]` . I think what @Paul_14 has in their config looks correct. I'd recommend using node names instead of IP addresses in this setting. However I do agree that the node logs are what we need to see to make further progress. ---
**Author:** ![Paul\_14](https://avatars.discourse-cdn.com/v4/letter/p/8edcca/32.png) [@Paul\_14](https://discuss.elastic.co/u/Paul_14)\ **Post date:** [June 12, 2019, 12:24pm UTC](https://discuss.elastic.co/t/nodes-not-communicating-in-3-node-cluster/185393/4 "2019-06-12T12:24:33Z")
Hi David & Cristiano, I made the change suggested by Cristiano, then restarted Elasticsearch. It didn't resolve the issue unfortunately... I reverted the change, and restarted again... The log output on restart is the same whether I use IP address or Node name for cluster.initial\_master\_nodes. Here is the log output after restarting: logs when using IP address instead of node name: ``` [2019-06-12T12:11:54,553][INFO][o.e.e.NodeEnvironment] [esp-data-0] using [1] data paths, mounts [[/datadisks/disk1 (/dev/sdc1)]], net usable_space [478.1gb], net total_space [503.8gb], types [ext4] [2019-06-12T12:11:54,558][INFO][o.e.e.NodeEnvironment] [esp-data-0] heap size [7.7gb], compressed ordinary object pointers [true] [2019-06-12T12:11:54,589][INFO][o.e.n.Node] [esp-data-0] node name [esp-data-0], node ID [IzZPSNlbSO2mNwtuPGSoOw] [2019-06-12T12:11:54,589][INFO][o.e.n.Node] [esp-data-0] version[7.0.0], pid[13269], build[default/deb/b7e28a7/2019-04-05T22:55:32.697037Z], OS[Linux/4.15.0-1045-azure/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12/12+33] [2019-06-12T12:11:54,590][INFO][o.e.n.Node] [esp-data-0] JVM home [/usr/share/elasticsearch/jdk] [2019-06-12T12:11:54,590][INFO][o.e.n.Node] [esp-data-0] JVM arguments [-Xms8009m, -Xmx8009m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-6877794265839147032, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Dio.netty.allocator.type=pooled, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true] [2019-06-12T12:11:56,587][INFO][o.e.p.PluginsService] [esp-data-0] loaded module [aggs-matrix-stats] **--- Loaded modules --- (removed this from logs to save space)** [2019-06-12T12:11:56,608][INFO][o.e.p.PluginsService] [esp-data-0] no plugins loaded [2019-06-12T12:12:01,337][INFO][o.e.x.m.p.l.CppLogMessageHandler] [esp-data-0] [controller/13373] [Main.cc@109] controller (64 bit): Version 7.0.0 (Build cdaa022645f38d) Copyright (c) 2019 Elasticsearch BV [2019-06-12T12:12:02,244][INFO][o.e.d.DiscoveryModule] [esp-data-0] using discovery type [zen] and seed hosts providers [settings] [2019-06-12T12:12:03,355][INFO][o.e.n.Node] [esp-data-0] initialized [2019-06-12T12:12:03,355][INFO][o.e.n.Node] [esp-data-0] starting ... [2019-06-12T12:12:03,495][INFO][o.e.t.TransportService] [esp-data-0] publish_address {192.248.16.7:9300}, bound_addresses {192.248.16.7:9300} [2019-06-12T12:12:03,503][INFO][o.e.b.BootstrapChecks] [esp-data-0] bound or publishing to a non-loopback address, enforcing bootstrap checks [2019-06-12T12:12:03,756][INFO][o.e.c.s.MasterService] [esp-data-0] elected-as-master ([1] nodes joined)[{esp-data-0}{IzZPSNlbSO2mNwtuPGSoOw}{FYG_9me0SWOfmDE2RtxG9w}{192.248.16.7}{192.248.16.7:9300}{ml.machine_memory=16796557312, xpack.installed=true, update_domain=1, ml.max_open_jobs=20, fault_domain=1} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 15, version: 108, reason: master node changed {previous [], current [{esp-data-0}{IzZPSNlbSO2mNwtuPGSoOw}{FYG_9me0SWOfmDE2RtxG9w}{192.248.16.7}{192.248.16.7:9300}{ml.machine_memory=16796557312, xpack.installed=true, update_domain=1, ml.max_open_jobs=20, fault_domain=1}]} [2019-06-12T12:12:03,977][INFO][o.e.c.s.ClusterApplierService] [esp-data-0] master node changed {previous [], current [{esp-data-0}{IzZPSNlbSO2mNwtuPGSoOw}{FYG_9me0SWOfmDE2RtxG9w}{192.248.16.7}{192.248.16.7:9300}{ml.machine_memory=16796557312, xpack.installed=true, update_domain=1, ml.max_open_jobs=20, fault_domain=1}]}, term: 15, version: 108, reason: Publication{term=15, version=108} [2019-06-12T12:12:04,033][INFO][o.e.h.AbstractHttpServerTransport] [esp-data-0] publish_address {192.248.16.7:9200}, bound_addresses {192.248.16.7:9200} [2019-06-12T12:12:04,033][INFO][o.e.n.Node] [esp-data-0] started [2019-06-12T12:12:04,286][INFO][o.e.l.LicenseService] [esp-data-0] license [786904f8-9db2-4771-a652-d920bd8db4ec] mode [trial] - valid [2019-06-12T12:12:04,295][INFO][o.e.g.GatewayService] [esp-data-0] recovered [6] indices into cluster_state [2019-06-12T12:12:05,046][INFO][o.e.c.r.a.AllocationService] [esp-data-0] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana_task_manager][0], [.kibana_1][0]] ...]). ``` ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 12, 2019, 12:40pm UTC](https://discuss.elastic.co/t/nodes-not-communicating-in-3-node-cluster/185393/5 "2019-06-12T12:40:51Z")
I think the first time you started these nodes up you did not have `cluster.initial_master_nodes` set, so [this note in the manual applies](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-bootstrap-cluster.html#modules-discovery-bootstrap-cluster-joining) to you, along with the advice to diagnose it for sure (each node reports a different cluster UUID) and the remedy (wipe the data and start again). If that's not the case then let us know and we'll dig deeper. ---
**Author:** ![Paul\_14](https://avatars.discourse-cdn.com/v4/letter/p/8edcca/32.png) [@Paul\_14](https://discuss.elastic.co/u/Paul_14)\ **Post date:** [June 12, 2019, 12:57pm UTC](https://discuss.elastic.co/t/nodes-not-communicating-in-3-node-cluster/185393/6 "2019-06-12T12:57:24Z")
Thanks very much David. This resolved the issue. ``` > curl 192.248.16.8:9200/_cat/nodes?v > ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name > 192.248.16.6 2 67 0 0.10 0.18 0.10 mdi * esp-data-1 > 192.248.16.8 3 68 0 0.06 0.13 0.08 mdi - esp-data-2 > 192.248.16.7 2 68 0 0.04 0.13 0.08 mdi - esp-data-0 ``` I had actually come across a similar instruction previously, however, when I went to /var/elasticsearch/data it was empty... It turns out that when deployed from Azure, by default **path.data** is set to **/datadisks/disk1/elasticsearch/data**. Deleting the data here then restarting resolved the problem.😀 Cheers, Paul ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 10, 2019, 1:01pm UTC](https://discuss.elastic.co/t/nodes-not-communicating-in-3-node-cluster/185393/7 "2019-07-10T13:01:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.