# Half-dead node lead to cluster hang
**URL:** https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658
**Category:** Elasticsearch
**Created:** [December 31, 2017, 1:42pm UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658 "2017-12-31T13:42:53Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@ginger](https://discuss.elastic.co/u/ginger)
#### Post date: [December 31, 2017, 1:42pm UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/1 "2017-12-31T13:42:53Z")
**Elasticsearch version (bin/elasticsearch --version):**
5.6.4
**JVM version (java -version):**
1.8.0\_91
**Description of the problem including expected versus actual behavior:**
In production enviroment, we have encounter hardware failure serveral times, which cause one or more nodes to half-dead, then the whole cluster hang.
**Elasticsearch cluster:**
3 nodes: 24 Cores, 128GB memory, 31GB heap
**Steps to reproduce:**
We use tc cmd to simulate the hardware failure and reproduce the problem:
1. start the cluster
2. do some heavy index(50%~ CPU)
3. use tc cmd to randomly drop packet:
```auto
tc qdisc add dev eth0 root netem loss 50%
```
Has anyone encountered similar problem? Any idea to tolerate such hardware failure?
Thanks : )
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [January 1, 2018, 12:03am UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/2 "2018-01-01T00:03:25Z")
What do you mean by "half-dead"? What do the logs show?
---
### Author:  [@ginger](https://discuss.elastic.co/u/ginger)
#### Post date: [January 2, 2018, 2:00am UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/3 "2018-01-02T02:00:25Z")
Thanks for your reply.
I mean that some machines or ethernet switches works abnormally. For example, those hardware can lost 50% network packet.
---
### Author:  [@ginger](https://discuss.elastic.co/u/ginger)
#### Post date: [January 29, 2018, 6:59am UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/4 "2018-01-29T06:59:38Z")
Any reply is appreciated.
Thanks
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [January 29, 2018, 7:05am UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/5 "2018-01-29T07:05:56Z")
If you have a non-reliable network, I am not sure what you can do?
---
### Author:  [@ginger](https://discuss.elastic.co/u/ginger)
#### Post date: [January 29, 2018, 7:12am UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/6 "2018-01-29T07:12:11Z")
In production clusters, this problem usually occurs when hardware failure. I just use tc cmd to simulate the hardware failure and reproduce the problem.
---
### Author:  [@ginger](https://discuss.elastic.co/u/ginger)
#### Post date: [February 1, 2018, 2:12am UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/7 "2018-02-01T02:12:13Z")
We update discovery.zen.fd.ping\_timeout configure to 2s and fix this problem for data node in 3-nodes test cluster. In my opinion, this is mainly because we can remove the data node from the cluster and reallocate shards as soon as possible.
But for active master node, this change does not work. The active master removed some datanodes and those data nodes come back soon. The active master does not die, and no new master is elected.
---
### Author:  [@ginger](https://discuss.elastic.co/u/ginger)
#### Post date: [February 1, 2018, 4:30am UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/8 "2018-02-01T04:30:07Z")
Any reply is appreciated!
---
### Author:  [@ginger](https://discuss.elastic.co/u/ginger)
#### Post date: [February 2, 2018, 11:46am UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/9 "2018-02-02T11:46:09Z")
I wonder that dose update discovery.zen.fd.ping\_timeout to 2s has some risk? Any advice on large scale cluster?
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [February 2, 2018, 8:51pm UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/10 "2018-02-02T20:51:35Z")
That is rather low, yes. Perhaps you can post your config and we can check it?
---
### Author:  [@ginger](https://discuss.elastic.co/u/ginger)
#### Post date: [February 3, 2018, 2:20am UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/11 "2018-02-03T02:20:18Z")
Our cluster has 100+ nodes, example config is as following:
```auto
cluster.name: es_xxx
node.data: true
node.ingest: true
node.master: false
node.name: data_node_1
path.data: ./data
processors: 16
indices.memory.index_buffer_size: 15%
node.attr.region: 99
node.attr.set: 25
node.attr.rack: 109699
node.attr.ip: {ip}
cluster.routing.allocation.awareness.attributes: ip
network.host: 0.0.0.0
network.publish_host: {ip}
http.port: 9201
transport.tcp.port: 9301
discovery.zen.ping.unicast.hosts: ["{ip1}:9301","{ip2}:9301","{ip3}:9301","{ip4}:9301","{ip5}:9301"]
discovery.zen.minimum_master_nodes: 3
bootstrap.seccomp: false
discovery.zen.fd.ping_timeout: 2s
```
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [February 3, 2018, 1:02pm UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/12 "2018-02-03T13:02:31Z")
Setting the ping timeout that low could cause a lot of problems as any long GC could cause the node to drop out. Sounds a bit risky to me, especially with a cluster that size.
What type of hardware failures are causing these problems? What type of hardware is the cluster deployed on?
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [February 3, 2018, 8:19pm UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/13 "2018-02-03T20:19:38Z")
> [@ginger](#):
>
> discovery.zen.minimum\_master\_nodes: 3
How many masters do you have?
> [@ginger](#):
>
> bootstrap.seccomp: false
Why are you doing this?
---
### Author:  [@ginger](https://discuss.elastic.co/u/ginger)
#### Post date: [February 4, 2018, 1:39pm UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/14 "2018-02-04T13:39:39Z")
```
What type of hardware is the cluster deployed on?
```
It's physical machines with local SSD disks.
```
What type of hardware failures are causing these problems?
```
One machine lost connection from other nodes or reboot. It's rather easy to use tc cmd to reproduce this problem in 3-nodes test cluster. In my opinion, the bad node isn't removed by the master node util 90s ping timeout, during which many bulk requests flood other nodes and cause old gc.
---
### Author:  [@ginger](https://discuss.elastic.co/u/ginger)
#### Post date: [February 4, 2018, 1:49pm UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/15 "2018-02-04T13:49:24Z")
> [@warkolm](#):
>
> How many masters do you have?
5 master nodes as the configure above.
> [@warkolm](#):
>
> bootstrap.seccomp: false
We know it's not secure, but we have some problems with this check.
---
### Author:  [@ginger](https://discuss.elastic.co/u/ginger)
#### Post date: [February 6, 2018, 1:57am UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/16 "2018-02-06T01:57:49Z")
> [@Christian\_Dahlqvist](#):
>
> Setting the ping timeout that low could cause a lot of problems as any long GC could cause the node to drop out.
This is exactly correct. We have noticed that there is some long gc(about 9s) in our product cluster. Setting the ping timeout that low is really a risky.
---
### Author:  [@ginger](https://discuss.elastic.co/u/ginger)
#### Post date: [February 7, 2018, 12:51am UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/17 "2018-02-07T00:51:36Z")
Any reply is appreciated!
---
### Author:  [@ywelsch](https://discuss.elastic.co/u/ywelsch)
#### Post date: [February 7, 2018, 10:00am UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/18 "2018-02-07T10:00:08Z")
It sounds like the network connection remains half-open (for causes, see e.g. [Detection of Half-Open (Dropped) Connections](https://blog.stephencleary.com/2009/05/detection-of-half-open-dropped.html)), i.e., the node fault detection on the master does not notice that the connection was closed. It will then take `discovery.zen.fd.ping_retries` (3) \* `discovery.zen.fd.ping_timeout` (30s) = 90 seconds to notice that the (data) node has become unavailable. Note that this is a rare event and usually indicates a hardware error. If 90 seconds is too long, you can lower those settings, with the risk that long garbage collection cycles can make your nodes being dropped by the master. Setting `discovery.zen.fd.ping_timeout` to 2s might be a bit too extreme, but values in the range of 5-10s (with 3 retries) should be ok.
> But for active master node, this change does not work. The active master removed some datanodes and those data nodes come back soon. The active master does not die, and no new master is elected.
Can you provide more information on this? How do these nodes come back? Can you provide logs from the active master node?
> We have noticed that there is some long gc(about 9s) in our product cluster. Setting the ping timeout that low is really a risky.
Have you investigated how to avoid those long GC cycles? What exactly is causing them? Is it due to client requests flooding the data nodes? Have you implement a client-side backoff strategy?
---
### Author:  [@ginger](https://discuss.elastic.co/u/ginger)
#### Post date: [February 7, 2018, 2:42pm UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/19 "2018-02-07T14:42:14Z")
> [@ywelsch](#):
>
> Setting discovery.zen.fd.ping\_timeout to 2s might be a bit too extreme, but values in the range of 5-10s (with 3 retries) should be ok.
We are testing it with discovery.zen.fd.ping\_timeout setting to 6s.
> [@ywelsch](#):
>
> Can you provide more information on this? How do these nodes come back? Can you provide logs from the active master node?
When using tc cmd to simulate hardware failure, the master node doesn't really die. The removed data nodes can come back through some alive data nodes. We will provide log soon.
> [@ywelsch](#):
>
> Have you investigated how to avoid those long GC cycles? What exactly is causing them? Is it due to client requests flooding the data nodes? Have you implement a client-side backoff strategy?
We indexed data heavily and about half heap memory is used(segment memory/bulk/search cache). We haven't implement a a client-side backoff strategy now.
---
### Author:  [@ginger](https://discuss.elastic.co/u/ginger)
#### Post date: [February 8, 2018, 2:31am UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/20 "2018-02-08T02:31:46Z")
> [@ywelsch](#):
>
> Can you provide logs from the active master node?
I send the master log via mail. The master log contains the online logs when one data-node machine failure.
[Next page](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658.md?page=2)