# What is the protocol used by elasticsearch for unicast zen discovery? **URL:** \ **Category:** Elasticsearch\ **Created:** [May 26, 2015, 10:53pm UTC](https://discuss.elastic.co/t/what-is-the-protocol-used-by-elasticsearch-for-unicast-zen-discovery/1344 "2015-05-26T22:53:56Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![ccrivelli](https://avatars.discourse-cdn.com/v4/letter/c/f1d935/32.png) [@ccrivelli](https://discuss.elastic.co/u/ccrivelli)\ **Post date:** [May 26, 2015, 10:53pm UTC](https://discuss.elastic.co/t/what-is-the-protocol-used-by-elasticsearch-for-unicast-zen-discovery/1344/1 "2015-05-26T22:53:56Z")
In the official doc they talk about ping. Are we talking about ICMP at network layer? Is there maybe TCP? What else? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 27, 2015, 12:43am UTC](https://discuss.elastic.co/t/what-is-the-protocol-used-by-elasticsearch-for-unicast-zen-discovery/1344/2 "2015-05-27T00:43:45Z")
The ping is TCP. ---
**Author:** ![ccrivelli](https://avatars.discourse-cdn.com/v4/letter/c/f1d935/32.png) [@ccrivelli](https://discuss.elastic.co/u/ccrivelli)\ **Post date:** [May 27, 2015, 1:06am UTC](https://discuss.elastic.co/t/what-is-the-protocol-used-by-elasticsearch-for-unicast-zen-discovery/1344/3 "2015-05-27T01:06:51Z")
Ping is ICMP, it's a protocol of the Network layer. It's not TCP. TCP is a protocol of the transport layer. So, are you saying that elasticsearch uses a 'ping check' at transport layer? Correct? Thanks ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 27, 2015, 1:08am UTC](https://discuss.elastic.co/t/what-is-the-protocol-used-by-elasticsearch-for-unicast-zen-discovery/1344/4 "2015-05-27T01:08:51Z")
In the context of ES pinging a node for discovery, that "ping" is done using TCP. I realise a literal ping is done via ICMP though. ---
**Author:** ![ccrivelli](https://avatars.discourse-cdn.com/v4/letter/c/f1d935/32.png) [@ccrivelli](https://discuss.elastic.co/u/ccrivelli)\ **Post date:** [May 27, 2015, 1:42am UTC](https://discuss.elastic.co/t/what-is-the-protocol-used-by-elasticsearch-for-unicast-zen-discovery/1344/5 "2015-05-27T01:42:28Z")
It makes sense, thanks. I'm still trying to figure out how to solve my 'failed to ping' issue. Actually I also notice that the cluster got stuck after running for a few hours and I see these errors in the logs: **failed to list shard** ```auto [2015-05-27 05:05:29,015][WARN][gateway.local] [hostneme148] [config][2]: failed to list shard state on node [TsCyLTuVT32aVt4TQH9HNg] org.elasticsearch.action.FailedNodeException: Failed node [TsCyLTuVT32aVt4TQH9HNg] at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.onFailure(TransportNodesOperationAction.java:206) at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.access$1000(TransportNodesOperationAction.java:97) at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction$4.handleException(TransportNodesOperationAction.java:178) at org.elasticsearch.transport.TransportService$3.run(TransportService.java:288) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.elasticsearch.transport.SendRequestTransportException: [hostname383][inet[/xx.xx.xx.34:9300]][internal:gateway/local/started_shards[n]] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:284) at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.start(TransportNodesOperationAction.java:165) at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.access$300(TransportNodesOperationAction.java:97) at org.elasticsearch.action.support.nodes.TransportNodesOperationAction.doExecute(TransportNodesOperationAction.java:70) at org.elasticsearch.action.support.nodes.TransportNodesOperationAction.doExecute(TransportNodesOperationAction.java:43) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:55) at org.elasticsearch.gateway.local.state.shards.TransportNodesListGatewayStartedShards.list(TransportNodesListGatewayStartedShards.java:66) at org.elasticsearch.gateway.local.LocalGatewayAllocator.buildShardStates(LocalGatewayAllocator.java:407) at org.elasticsearch.gateway.local.LocalGatewayAllocator.allocateUnassigned(LocalGatewayAllocator.java:127) at org.elasticsearch.cluster.routing.allocation.allocator.ShardsAllocators.allocateUnassigned(ShardsAllocators.java:74) at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:219) at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:162) at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:148) at org.elasticsearch.discovery.zen.ZenDiscovery$6.execute(ZenDiscovery.java:563) at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:365) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:188) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:158) ... 3 more ``` **NodeNotConnectedException** ```auto Caused by: org.elasticsearch.transport.NodeNotConnectedException: [hostname383][inet[/xx.xx.xx.34:9300]] Node not connected at org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:936) at org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:629) at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:270) ... 20 more [2015-05-27 05:05:29,017][WARN][gateway.local] [hostname148] [checks][2]: failed to list shard state on node [TsCyLTuVT32aVt4TQH9HNg] org.elasticsearch.action.FailedNodeException: Failed node [TsCyLTuVT32aVt4TQH9HNg] at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.onFailure(TransportNodesOperationAction.java:206) at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.access$1000(TransportNodesOperationAction.java:97) at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction$4.handleException(TransportNodesOperationAction.java:178) at org.elasticsearch.transport.TransportService$3.run(TransportService.java:288) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.elasticsearch.transport.SendRequestTransportException: [hostname383][inet[/10.38.213.34:9300]][internal:gateway/local/started_shards[n]] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:284) at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.start(TransportNodesOperationAction.java:165) at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.access$300(TransportNodesOperationAction.java:97) at org.elasticsearch.action.support.nodes.TransportNodesOperationAction.doExecute(TransportNodesOperationAction.java:70) at org.elasticsearch.action.support.nodes.TransportNodesOperationAction.doExecute(TransportNodesOperationAction.java:43) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:55) at org.elasticsearch.gateway.local.state.shards.TransportNodesListGatewayStartedShards.list(TransportNodesListGatewayStartedShards.java:66) at org.elasticsearch.gateway.local.LocalGatewayAllocator.buildShardStates(LocalGatewayAllocator.java:407) at org.elasticsearch.gateway.local.LocalGatewayAllocator.allocateUnassigned(LocalGatewayAllocator.java:127) at org.elasticsearch.cluster.routing.allocation.allocator.ShardsAllocators.allocateUnassigned(ShardsAllocators.java:74) at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:219) at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:162) at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:148) at org.elasticsearch.discovery.zen.ZenDiscovery$6.execute(ZenDiscovery.java:563) at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:365) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:188) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:158) ... 3 more Caused by: org.elasticsearch.transport.NodeNotConnectedException: [hostname383][inet[/xx.xx.xx.34:9300]] Node not connected at org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:936) at org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:629) at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:270) ... 20 more ``` **transport disconnected** ```auto [2015-05-27 05:05:29,019][INFO][cluster.service] [hostname148] removed {[hostname036][1bSE8cf5Sw6FlXbyqR3ocQ][hostname036][inet[/xx.xx.xx.137:9300]]{data=false, master=false},}, reason: zen-disco-node_failed([hostname036][1bSE8cf5Sw6FlXbyqR3ocQ][nhostname036][inet[/xx.xx.xx.137:9300]]{data=false, master=false}), reason transport disconnected [2015-05-27 05:05:29,053][INFO][cluster.service] [hostname148] removed {[hostname383][TsCyLTuVT32aVt4TQH9HNg][hostname383][inet[/xx.xx.xx.34:9300]],}, reason: zen-disco-node_failed([hostname383][TsCyLTuVT32aVt4TQH9HNg][hostname383][inet[/xx.xx.xx.34:9300]]), reason transport disconnected ``` I found something similar here: **ES 1.4.2 random node disconnect ยท Issue #9212** - [https://github.com/elastic/elasticsearch/issues/9212](https://github.com/elastic/elasticsearch/issues/9212) but I haven't overloaded my cluster yet and It's running elasticsearch 1.5, not 1.4.2. Do you (or someone else) have some some advice to address me in order to solve the issue? Any thoughts? Thanks always ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 27, 2015, 2:51am UTC](https://discuss.elastic.co/t/what-is-the-protocol-used-by-elasticsearch-for-unicast-zen-discovery/1344/6 "2015-05-27T02:51:58Z")
Can you show us your config (minus comments/empty lines please)? Are the hosts in the same DC? Are you measuring network latency, or your cluster and it's nodes using Marvel or similar? ---
**Author:** ![ccrivelli](https://avatars.discourse-cdn.com/v4/letter/c/f1d935/32.png) [@ccrivelli](https://discuss.elastic.co/u/ccrivelli)\ **Post date:** [May 27, 2015, 6:36am UTC](https://discuss.elastic.co/t/what-is-the-protocol-used-by-elasticsearch-for-unicast-zen-discovery/1344/7 "2015-05-27T06:36:43Z")
Sure, I have 5 master/data nodes + 2 client nodes (just es load balancers). This is my config: ```auto cluster.name: elasticsearch_dev node.name: "" path.data: /esdata1, /esdata2 discovery.zen.minimum_master_nodes: 1 gateway.recover_after_nodes: 3 gateway.recover_after_time: 5m gateway.expected_nodes: 5 discovery.zen.ping.multicast.enabled: false ## fqdn everywhere discovery.zen.ping.unicast.hosts: ["hostname272:9300", "hostname383:9300", "hostname587:9300", "hostname148:9300", "hostname038:9300","hostname036:9300","hostname108:9300"] index.number_of_replicas: 2 discovery.zen.ping.timeout: 30s bootstrap.mlockall: true ## added later but same behavior # Add fault detection discovery.zen.fd.ping_interval: 15s discovery.zen.fd.ping_timeout: 60s discovery.zen.fd.ping_retries: 5 ``` **\_nodes** [http://pastebin.com/iwZ1Z6na](http://pastebin.com/iwZ1Z6na) **\_nodes/stats** [http://pastebin.com/NuQ2Gxjk](http://pastebin.com/NuQ2Gxjk) The hosts are in the same DC. They are virtual machines (probaby on the same physical server, but I'm not sure of it). I installed Marvel, but I got additional ERRORS in the logs, then I uninstalled it. At the moment I'm checking the network running continuously from each master/data node: 1. a fping to the other hosts every 3 seconds (ICMP ping) and all the hosts are always alive ```auto Wed May 27 16:12:38 AEST 2015 -- Sleep -- hostname272 is alive hostname383 is alive hostname587 is alive hostname148 is alive hostname038 is alive hostname036 is alive hosntame108 is alive Wed May 27 16:12:41 AEST 2015 -- Sleep -- hostname272 is alive hostname383 is alive hostname587 is alive hostname148 is alive hostname038 is alive hostname036 is alive hosntame108 is alive ``` 1. a hping to the other hosts o port 9300 (1 TCP Syn Packet every 2 seconds) but this check is causing another exception on es cluster, so I will stop doing it: `StreamCorruptedException: invalid internal transport message format, got (47,45,54,20)` If there is something other I could check, please let me know. Thanks ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:11am UTC](https://discuss.elastic.co/t/what-is-the-protocol-used-by-elasticsearch-for-unicast-zen-discovery/1344/8 "2017-07-06T00:11:44Z")