# Cluster Hangs for 20 seconds, on a single node crush
**URL:** https://discuss.elastic.co/t/cluster-hangs-for-20-seconds-on-a-single-node-crush/198191
**Category:** Elasticsearch
**Created:** [September 5, 2019, 7:53am UTC](https://discuss.elastic.co/t/cluster-hangs-for-20-seconds-on-a-single-node-crush/198191 "2019-09-05T07:53:04Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@ran\_n](https://discuss.elastic.co/u/ran_n)
#### Post date: [September 5, 2019, 7:53am UTC](https://discuss.elastic.co/t/cluster-hangs-for-20-seconds-on-a-single-node-crush/198191/1 "2019-09-05T07:53:04Z")
Hi,
I have a simple setup, with test data of 10 documents.
I have 3 nodes, 2 data 1 only master
I have 5 shards with 1 replica.
I run a search query every second via small simulator
I then disable the network card on the node that contains only the replicas.
My search queries are lagging - all of them, during the first 20 seconds post the card disable
So first call post the NIC down scenario, will get reply after 19s
Second call will get reply after 18s
Third call will get reply after 17s
I am using Elastic 6.7.1 - can someone elaborate on the root cause for this ?
How comes that killing 1 node my cluster hangs for 20 seconds ?
Thanks in advance,
Ran
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [September 5, 2019, 8:23am UTC](https://discuss.elastic.co/t/cluster-hangs-for-20-seconds-on-a-single-node-crush/198191/2 "2019-09-05T08:23:37Z")
How have you configured `/proc/sys/net/ipv4/tcp_retries2`? It defaults to 15 which is far too many IMO, and there are [others who recommend reducing it to 3](https://access.redhat.com/solutions/726753) for high-availability situations.
There's also an issue in older Elasticsearch versions (fixed in [#39629](https://github.com/elastic/elasticsearch/pull/39629), released in 7.2.0) that could slow down cluster state updates in your situation. I don't know that this will affect this experiment, unless you're disabling the NIC on the master, but I recommend upgrading to a later version.
---
### Author:  [@ran\_n](https://discuss.elastic.co/u/ran_n)
#### Post date: [September 5, 2019, 8:39am UTC](https://discuss.elastic.co/t/cluster-hangs-for-20-seconds-on-a-single-node-crush/198191/3 "2019-09-05T08:39:04Z")
Thanks ! Will try to repeat the test on the latest...
I didn't mention I test it under windows boxes
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [September 5, 2019, 8:41am UTC](https://discuss.elastic.co/t/cluster-hangs-for-20-seconds-on-a-single-node-crush/198191/4 "2019-09-05T08:41:24Z")
Ah, ok, I think Windows has a similar kind of parameter to control TCP retries, but I don't know what it is.
---
### Author:  [@ran\_n](https://discuss.elastic.co/u/ran_n)
#### Post date: [September 5, 2019, 11:15am UTC](https://discuss.elastic.co/t/cluster-hangs-for-20-seconds-on-a-single-node-crush/198191/5 "2019-09-05T11:15:20Z")
Tested it on the latest, problem persists...
---
### Author:  [@ran\_n](https://discuss.elastic.co/u/ran_n)
#### Post date: [September 5, 2019, 11:21am UTC](https://discuss.elastic.co/t/cluster-hangs-for-20-seconds-on-a-single-node-crush/198191/6 "2019-09-05T11:21:39Z")
In general I don't understand the flow regardless of TCP/ linux.
Node A contain the primary shard
Nove B containg copy shard NIC is disabled
Why Node A is hanging for 20 seconds when I am searching data contained on its shards ?!
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [September 5, 2019, 11:27am UTC](https://discuss.elastic.co/t/cluster-hangs-for-20-seconds-on-a-single-node-crush/198191/7 "2019-09-05T11:27:52Z")
> [@ran\_n](#):
>
> Why Node A is hanging for 20 seconds when I am searching data contained on its shards ?!
Normally a search will be distributed across the whole cluster, so I would expect it to try and search some of the shards on node B. If your OS is configured to retry transmission an unreasonable number of times before giving up then those remote searches could take a long time to fail.
---
### Author:  [@ran\_n](https://discuss.elastic.co/u/ran_n)
#### Post date: [September 5, 2019, 11:39am UTC](https://discuss.elastic.co/t/cluster-hangs-for-20-seconds-on-a-single-node-crush/198191/8 "2019-09-05T11:39:04Z")
I understand ES is round robin between the nodes, but I make a call every sec - all of the calls are hanged during this time.
Even if ES is distributing my search the local shard should reply and I expect to get the reply back.
I made this test with only one document in my index... latest code and still issue occur.
Please note that the test is disabling the NIC, if I kill the service all works perfect without this hang...
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [September 5, 2019, 11:47am UTC](https://discuss.elastic.co/t/cluster-hangs-for-20-seconds-on-a-single-node-crush/198191/9 "2019-09-05T11:47:17Z")
> [@ran\_n](#):
>
> I understand ES is round robin between the nodes, but I make a call every sec - all of the calls are hanged during this time.
I think you misunderstand. **Each** search is distributed across the cluster, and is expected to involve the disconnected node.
---
### Author:  [@ran\_n](https://discuss.elastic.co/u/ran_n)
#### Post date: [September 5, 2019, 12:13pm UTC](https://discuss.elastic.co/t/cluster-hangs-for-20-seconds-on-a-single-node-crush/198191/10 "2019-09-05T12:13:07Z")
I don't understand the logic in this design :
ES sends my search query to all nodes, lets say I have 5 nodes, where one of the has crushed.
Now I am getting replies from 4 nodes but instead of returning the results, the server will wait for the reply from node #5 that is down?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [September 5, 2019, 12:22pm UTC](https://discuss.elastic.co/t/cluster-hangs-for-20-seconds-on-a-single-node-crush/198191/11 "2019-09-05T12:22:38Z")
Right. The much more common case is that you _don't_ have a failing node and there you want each search to use all the CPU/IO/etc. resources in the cluster, rather than restricting itself to a single node.
Elasticsearch will notice that the remote node is down as soon as the OS tells it the connection has dropped. The issue you're facing is that the OS is taking far longer than you would like to notice that the connection has dropped.
---
### Author:  [@ran\_n](https://discuss.elastic.co/u/ran_n)
#### Post date: [September 5, 2019, 12:27pm UTC](https://discuss.elastic.co/t/cluster-hangs-for-20-seconds-on-a-single-node-crush/198191/12 "2019-09-05T12:27:37Z")
🙂 Thanks for being patient...
Sending to several nodes makes sense, np
But why should server wait for all nodes to reply - why not return the reply once one of the node replied ?
BTW - can you try and direct me to the code that is in charge of this part of the flow ?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [September 5, 2019, 12:36pm UTC](https://discuss.elastic.co/t/cluster-hangs-for-20-seconds-on-a-single-node-crush/198191/13 "2019-09-05T12:36:05Z")
> [@ran\_n](#):
>
> But why should server wait for all nodes to reply - why not return the reply once one of the node replied ?
It only searches one copy of each shard, so it needs to collect all the responses (or failures) before it can respond.
> [@ran\_n](#):
>
> BTW - can you try and direct me to the code that is in charge of this part of the flow ?
It's hard to point at any one place that implements all this behaviour (it's actually quite complicated) but maybe this is a useful starting point?
>
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 3, 2019, 12:45pm UTC](https://discuss.elastic.co/t/cluster-hangs-for-20-seconds-on-a-single-node-crush/198191/14 "2019-10-03T12:45:09Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.