# Simulating network connect failure between nodes

**URL:** <https://discuss.elastic.co/t/simulating-network-connect-failure-between-nodes/67657>\
**Category:** Elasticsearch\
**Created:** [November 30, 2016, 5:29pm UTC](https://discuss.elastic.co/t/simulating-network-connect-failure-between-nodes/67657 "2016-11-30T17:29:43Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![waqark3389](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/waqark3389/32/34265_2.png) [@waqark3389](https://discuss.elastic.co/u/waqark3389)\
**Post date:** [November 30, 2016, 5:29pm UTC](https://discuss.elastic.co/t/simulating-network-connect-failure-between-nodes/67657/1 "2016-11-30T17:29:43Z")

</div>

I have 5 nodes, all eligible as master and all data nodes.discovery.zen.minimum\_master\_nodes is set to 3:

node1,node2,node3,node4 and node5. Everything else is default.

So I tried to simulate a network problem between node5 and the cluster. So I blocked (via firewall) outgoing from node5 to all other nodes and blocked incoming to node5 from the other nodes.  
I should see node5 has been removed from cluster but instead I cannot do any POST/GET to ANY node in the cluster. On the master, node3, I can see timeouts connecting to node5.

Querying any node including node5 (the isolated node) e.g. node1:9200/\_cluster/health is retuning green and it thinks it has 5 nodes. After about 30 minutes the nodes finally realise node5 is not responding and they remove him from the pool.

The whole cluster went down for over 30 minutes in one node is isolated. I can see the following on node3 logs repeated every minute, who is master at this time:

[node3] failed to execute on node [VdSa2w0tSwiUyNNFhzvNXg]  
org.elasticsearch.transport.NodeDisconnectedException: [node5][172.16.99.234:9300][cluster:monitor/nodes/stats[n]] disconnected

Why is it the cluster is unusable when a node is isolated? What can I do to speed up the process of recovery?

Here is the config which is the same on all 5 nodes except name:

> cluster.name: elasticsearchTestCluster  
> node.name: node3  
> http.cors.enabled: true  
> http.cors.allow-origin: "\*"  
> network.host: 0.0.0.0

> discovery.zen.ping.unicast.hosts: ["172.16.99.230", "172.16.99.231", "172.16.99.232", "172.16.99.233", "172.16.99.234"]  
> discovery.zen.minimum\_master\_nodes: 3

> path.data: /home/elasticsearchData

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 4, 2016, 6:29am UTC](https://discuss.elastic.co/t/simulating-network-connect-failure-between-nodes/67657/2 "2016-12-04T06:29:25Z")

</div>

Thanks for reporting.

Can you add here the cluster state before the node is isolated and after.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 1, 2017, 6:29am UTC](https://discuss.elastic.co/t/simulating-network-connect-failure-between-nodes/67657/3 "2017-01-01T06:29:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.