# Cluster nodes unable to communicate with each other **URL:** \ **Category:** Elasticsearch\ **Created:** [April 12, 2019, 2:18pm UTC](https://discuss.elastic.co/t/cluster-nodes-unable-to-communicate-with-each-other/176644 "2019-04-12T14:18:11Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![rao-uno](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@rao-uno](https://discuss.elastic.co/u/rao-uno)\ **Post date:** [April 12, 2019, 2:18pm UTC](https://discuss.elastic.co/t/cluster-nodes-unable-to-communicate-with-each-other/176644/1 "2019-04-12T14:18:11Z")
Hi, I have a cluster of 3 nodes. The nodes storage got full (91%) and no new index were being created. But then my servers hanged and had to be rebooted. Now when I am starting the cluster, the cluster is not able to identify any other nodes so all my indices' health status is **RED**. The storage space is still full and there are unassigned replicas. Can anyone please help me in getting my cluster communication. There has been no change in host names and IPs. Thank you ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 12, 2019, 2:34pm UTC](https://discuss.elastic.co/t/cluster-nodes-unable-to-communicate-with-each-other/176644/2 "2019-04-12T14:34:16Z")
`cluster.routing.allocation.disk.watermark.high` defaults to `90%`, and if a node is above this level then it will not allocate any primaries. As a short-term fix you can increase `cluster.routing.allocation.disk.watermark.high` to, say, `92%`, but you will need to free up some space or purchase some more storage in the very near future to resolve this properly. ---
**Author:** ![rao-uno](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@rao-uno](https://discuss.elastic.co/u/rao-uno)\ **Post date:** [April 12, 2019, 6:09pm UTC](https://discuss.elastic.co/t/cluster-nodes-unable-to-communicate-with-each-other/176644/3 "2019-04-12T18:09:22Z")
Thank you David, but I had a doubt, would this allow my nodes to be able to join and form the cluster? Currently my cluster status only shows one node (that node itself) on the 3 elasticsearch nodes. Is the high disk watermark not allowing them to form the cluster? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 12, 2019, 6:50pm UTC](https://discuss.elastic.co/t/cluster-nodes-unable-to-communicate-with-each-other/176644/4 "2019-04-12T18:50:15Z")
No, the disk watermark wouldn't be doing that. Do you have `discovery.zen.minimum_master_nodes` set to 2 on every node? ---
**Author:** ![rao-uno](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@rao-uno](https://discuss.elastic.co/u/rao-uno)\ **Post date:** [April 13, 2019, 8:33am UTC](https://discuss.elastic.co/t/cluster-nodes-unable-to-communicate-with-each-other/176644/5 "2019-04-13T08:33:57Z")
Yes it was set to 2, but I kept getting error , Not enough master nodes discovered during pinging. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 13, 2019, 8:51am UTC](https://discuss.elastic.co/t/cluster-nodes-unable-to-communicate-with-each-other/176644/6 "2019-04-13T08:51:23Z")
Can you share the logs from all the nodes? Without more information we can only really speculate. ---
**Author:** ![rao-uno](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@rao-uno](https://discuss.elastic.co/u/rao-uno)\ **Post date:** [April 14, 2019, 7:27am UTC](https://discuss.elastic.co/t/cluster-nodes-unable-to-communicate-with-each-other/176644/7 "2019-04-14T07:27:44Z")
These are some of the logs from my node ```auto [2019-04-12T10:54:46,763][DEBUG][o.e.a.ActionModule] [elk-node-1] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security [2019-04-12T10:54:47,798][INFO][o.e.d.DiscoveryModule] [elk-node-1] using discovery type [zen] and host providers [settings] [2019-04-12T10:54:49,784][INFO][o.e.n.Node] [elk-node-1] initialized [2019-04-12T10:54:49,785][INFO][o.e.n.Node] [elk-node-1] starting ... [2019-04-12T10:54:50,186][INFO][o.e.t.TransportService] [elk-node-1] publish_address {xx.xx.xx.xx:9300}, bound_addresses {[::]:9300} [2019-04-12T10:54:50,401][INFO][o.e.b.BootstrapChecks] [elk-node-1] bound or publishing to a non-loopback address, enforcing bootstrap checks [2019-04-12T10:54:53,509][WARN][o.e.d.z.ZenDiscovery] [elk-node-1] not enough master nodes discovered during pinging (found [[Candidate{node={elk-node-1}{9uVegm9tSyG2HiObIzzZzw}{UENmmjH2TrGS-EfiVK3Lhg}{xx.xx.xx.xx}{xx.xx.xx.xx:9300}{ml.machine_memory=8186187776, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-04-12T10:54:56,513][WARN][o.e.d.z.ZenDiscovery] [elk-node-1] not enough master nodes discovered during pinging (found [[Candidate{node={elk-node-1}{9uVegm9tSyG2HiObIzzZzw}{UENmmjH2TrGS-EfiVK3Lhg}{xx.xx.xx.xx}{xx.xx.xx.xx:9300}{ml.machine_memory=8186187776, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-04-12T10:54:59,516][WARN][o.e.d.z.ZenDiscovery] [elk-node-1] not enough master nodes discovered during pinging (found [[Candidate{node={elk-node-1}{9uVegm9tSyG2HiObIzzZzw}{UENmmjH2TrGS-EfiVK3Lhg}{xx.xx.xx.xx}{xx.xx.xx.xx:9300}{ml.machine_memory=8186187776, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-04-12T10:55:02,520][WARN][o.e.d.z.ZenDiscovery] [elk-node-1] not enough master nodes discovered during pinging (found [[Candidate{node={elk-node-1}{9uVegm9tSyG2HiObIzzZzw}{UENmmjH2TrGS-EfiVK3Lhg}{xx.xx.xx.xx}{xx.xx.xx.xx:9300}{ml.machine_memory=8186187776, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-04-12T10:55:05,525][WARN][o.e.d.z.ZenDiscovery] [elk-node-1] not enough master nodes discovered during pinging (found [[Candidate{node={elk-node-1}{9uVegm9tSyG2HiObIzzZzw}{UENmmjH2TrGS-EfiVK3Lhg}{xx.xx.xx.xx}{xx.xx.xx.xx:9300}{ml.machine_memory=8186187776, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-04-12T10:55:35,554][WARN][o.e.d.z.ZenDiscovery] [elk-node-1] not enough master nodes discovered during pinging (found [[Candidate{node={elk-node-1}{9uVegm9tSyG2HiObIzzZzw}{UENmmjH2TrGS-EfiVK3Lhg}{xx.xx.xx.xx}{xx.xx.xx.xx:9300}{ml.machine_memory=8186187776, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-04-12T10:55:35,948][WARN][r.suppressed] [elk-node-1] path: /.reporting-*/esqueue/_search, params: {index=.reporting-*, type=esqueue, version=true} org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized]; at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:166) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:152) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:297) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.action.search.TransportSearchAction.lambda$doExecute$4(TransportSearchAction.java:193) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.5.4.jar:6.5.4] ``` ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 14, 2019, 7:31am UTC](https://discuss.elastic.co/t/cluster-nodes-unable-to-communicate-with-each-other/176644/8 "2019-04-14T07:31:49Z")
Thanks @rao-uno, it looks like the nodes just don't know about each other. Have you configured `discovery.zen.ping.unicast.hosts` correctly on each node? It should contain the hostnames or IP addresses of each master-eligible node in the cluster. ---
**Author:** ![rao-uno](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@rao-uno](https://discuss.elastic.co/u/rao-uno)\ **Post date:** [April 14, 2019, 7:36am UTC](https://discuss.elastic.co/t/cluster-nodes-unable-to-communicate-with-each-other/176644/9 "2019-04-14T07:36:36Z")
Yes, I have not made any changes to my config files after the reboot and they have the correct host IPs. I even checked `netstat -plnt` to check the ports being listened to and found `9200` and `9300` ports listed there. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 14, 2019, 7:59am UTC](https://discuss.elastic.co/t/cluster-nodes-unable-to-communicate-with-each-other/176644/10 "2019-04-14T07:59:55Z")
Ok, this is strange. Could you set `logger.org.elasticsearch.discovery.zen.UnicastZenPing: TRACE` and restart a node to get more detail on what's going wrong with discovery? ---
**Author:** ![rao-uno](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@rao-uno](https://discuss.elastic.co/u/rao-uno)\ **Post date:** [April 14, 2019, 8:29am UTC](https://discuss.elastic.co/t/cluster-nodes-unable-to-communicate-with-each-other/176644/11 "2019-04-14T08:29:13Z")
Okay sure. I will set this configuration and get more details on it by tomorrow since I don't have access to the nodes now. ---
**Author:** ![rao-uno](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@rao-uno](https://discuss.elastic.co/u/rao-uno)\ **Post date:** [April 15, 2019, 7:58am UTC](https://discuss.elastic.co/t/cluster-nodes-unable-to-communicate-with-each-other/176644/12 "2019-04-15T07:58:58Z")
> [@DavidTurner](#): > > `logger.org.elasticsearch.discovery.zen.UnicastZenPing: TRACE` How do I set this configuration? Do I add this in elasticsearch.yml or is there any API for it? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 15, 2019, 8:01am UTC](https://discuss.elastic.co/t/cluster-nodes-unable-to-communicate-with-each-other/176644/13 "2019-04-15T08:01:02Z")
Yes, put that line in `elasticsearch.yml`. There [is also an API](https://www.elastic.co/guide/en/elasticsearch/reference/7.0/logging.html#configuring-logging-levels), but it only works after the cluster has formed so it's of no use to you ☹ ---
**Author:** ![rao-uno](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@rao-uno](https://discuss.elastic.co/u/rao-uno)\ **Post date:** [April 15, 2019, 8:34am UTC](https://discuss.elastic.co/t/cluster-nodes-unable-to-communicate-with-each-other/176644/14 "2019-04-15T08:34:27Z")
```auto [2019-04-15T13:53:31,026][TRACE][o.e.d.z.UnicastZenPing] [elk-node-1] [31] failed to ping {xx.xx.xx.xx:9300}{gs3ICmKjS06xWT4TrJ2rIw}{xx.xx.xx.xx}{xx.xx.xx.xx:9300} org.elasticsearch.transport.ConnectTransportException: [][xx.xx.xx.xx:9300] connect_exception at org.elasticsearch.transport.TcpChannel.awaitConnected(TcpChannel.java:165) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:454) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:117) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.transport.ConnectionManager.internalOpenConnection(ConnectionManager.java:237) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.transport.ConnectionManager.openConnection(ConnectionManager.java:95) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.transport.TransportService.openConnection(TransportService.java:393) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.discovery.zen.UnicastZenPing$PingingRound.getOrConnect(UnicastZenPing.java:364) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.discovery.zen.UnicastZenPing$3.doRun(UnicastZenPing.java:471) [elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.4.jar:6.5.4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_102] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102] Caused by: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: No route to host: /xx.xx.xx.xx:9300 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?] at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?] at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) ~[?:?] ... 1 more Caused by: java.net.NoRouteToHostException: No route to host at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?] at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?] at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) ~[?:?] ``` This is the error I'm getting with the TRACE option. I have sanitized the logs(replaced my IP with 'x'). I pinged the server (without the port) and was able to ping it. I have one doubt, the port is listed as`tcp6` in the `netstat -plnt` output, is this correct? And am I missing something else in the logs? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 15, 2019, 9:02am UTC](https://discuss.elastic.co/t/cluster-nodes-unable-to-communicate-with-each-other/176644/15 "2019-04-15T09:02:48Z")
Ok, this is strange. `No route to host` is a pretty low-level error indicating something is very wrong at the network level, but I wouldn't normally expect a `ping` to work in the presence of this error either. Can you double (triple) check that the IP address is right? `ping` does something quite different from what Elasticsearch is trying to do; `curl http://xx.xx.xx.xx:9300/` is closer, and should yield `This is not an HTTP port` if successful. The next thing I think I'd try is using `tcpdump` to get a packet capture during a pinging round in order to see exactly what's going on with the network: ```plaintext sudo tcpdump -i -s 65535 -w capture-$(date +%s).cap ``` You can then open the capture file in [Wireshark](https://www.wireshark.org), or you can send it to me at [david.turner@elastic.co](mailto:david.turner@elastic.co) and I'll take a look. > [@rao-uno](#): > > I have one doubt, the port is listed as `tcp6` in the `netstat -plnt` output, is this correct? I think that's ok, see [this StackOverflow answer](https://unix.stackexchange.com/a/237747) for example. ---
**Author:** ![rao-uno](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@rao-uno](https://discuss.elastic.co/u/rao-uno)\ **Post date:** [April 15, 2019, 10:07am UTC](https://discuss.elastic.co/t/cluster-nodes-unable-to-communicate-with-each-other/176644/16 "2019-04-15T10:07:54Z")
Hi David, I'm getting `Failed to connect to xx.xx.xx.xx:9300; No route to host` when I'm running `curl http://xx.xx.xx.xx:9300/` ---
**Author:** ![rao-uno](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@rao-uno](https://discuss.elastic.co/u/rao-uno)\ **Post date:** [April 15, 2019, 10:12am UTC](https://discuss.elastic.co/t/cluster-nodes-unable-to-communicate-with-each-other/176644/17 "2019-04-15T10:12:55Z")
The tcpdump is giving `Destination unreachable (Host administratively prohibited)` for the cluster servers ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 15, 2019, 10:24am UTC](https://discuss.elastic.co/t/cluster-nodes-unable-to-communicate-with-each-other/176644/18 "2019-04-15T10:24:46Z")
Ok, that sounds like a firewall rule is preventing these nodes from communicating. For instance there is an [`iptables`](https://linux.die.net/man/8/iptables) rule set up to reject this traffic with `-j REJECT --reject-with icmp-host-prohibited`. ---
**Author:** ![rao-uno](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@rao-uno](https://discuss.elastic.co/u/rao-uno)\ **Post date:** [April 15, 2019, 10:37am UTC](https://discuss.elastic.co/t/cluster-nodes-unable-to-communicate-with-each-other/176644/19 "2019-04-15T10:37:16Z")
Yes, I do see this mentioned in `iptables -L`. `REJECT all -- anywhere anywhere reject-with icmp-host-prohibited` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 13, 2019, 10:37am UTC](https://discuss.elastic.co/t/cluster-nodes-unable-to-communicate-with-each-other/176644/20 "2019-05-13T10:37:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.