# Half-dead node lead to cluster hang **URL:** \ **Category:** Elasticsearch\ **Created:** [December 31, 2017, 1:42pm UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658 "2017-12-31T13:42:53Z")\ **Posts on this page:** 13\ **Page:** 2
**Author:** ![ginger](https://avatars.discourse-cdn.com/v4/letter/g/34f0e0/32.png) [@ginger](https://discuss.elastic.co/u/ginger)\ **Post date:** [February 8, 2018, 3:35am UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/21 "2018-02-08T03:35:49Z")
BTW, some similar phenomenons as [https://github.com/elastic/elasticsearch/issues/27194:](https://github.com/elastic/elasticsearch/issues/27194:) 1. Through we only have 31shards in test cluster, we observe a lot of shard failure tasks: 225 _add\_listener_ 1 cluster\_reroute(async\_shard\_fetch) 1 delayed\_allocation\_reroute 6 master 106945 shard-failed 1 zen-disco-node-failed 21 zen-disco-node-join 2. Killing a node normally does not reproduce the problem. The recovery in that scenario is normal, a reasonable amount of shard-failed tasks and quick recovery. ---
**Author:** ![ywelsch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ywelsch/32/7751_2.png) [@ywelsch](https://discuss.elastic.co/u/ywelsch)\ **Post date:** [February 8, 2018, 4:29pm UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/22 "2018-02-08T16:29:54Z")
> [@ginger](#): > > I send the master log via mail. I did not receive those logs. Please send them to my [elastic.co](http://elastic.co) e-mail address (yannick@...). ---
**Author:** ![ginger](https://avatars.discourse-cdn.com/v4/letter/g/34f0e0/32.png) [@ginger](https://discuss.elastic.co/u/ginger)\ **Post date:** [February 9, 2018, 1:20am UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/23 "2018-02-09T01:20:50Z")
Sorry for sending a wrong mail address. I send it again. ---
**Author:** ![ginger](https://avatars.discourse-cdn.com/v4/letter/g/34f0e0/32.png) [@ginger](https://discuss.elastic.co/u/ginger)\ **Post date:** [February 9, 2018, 12:01pm UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/24 "2018-02-09T12:01:47Z")
@ywelsch We try to refuse bulk request after high JVM old generation usage in a plugin. The test cluster won't hang, but we observed an interesting problem, which may be the real problem: 5 minutes after the above tc cmd, many transport\_server\_worker threads of the master is blocked by ShardFailedTransportHandler.messageReceived() method, and sometimes call of the master transport address cost 10+s. We wonder that the master of the online cluster, whose log is provided before, was also blocked by this problem, that's why a new master was elected. The full jstack: > ---
**Author:** ![ywelsch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ywelsch/32/7751_2.png) [@ywelsch](https://discuss.elastic.co/u/ywelsch)\ **Post date:** [February 9, 2018, 2:10pm UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/25 "2018-02-09T14:10:21Z")
That's definitely an interesting theory. The stack trace indicates that the amount of "shard failed" events that are handled on the master look to overwhelm it and block network threads. The logs from the production cluster are inconclusive, they show that the master node has trouble reconnecting to the rest of the cluster, but there are also minutes going by without any log information. In case where this happens again in production, can you take stack dumps / heap dumps of the master? Can you also provide more information on all the steps you took to reproduce this, in particular the ones which led to the 106945 shard-failed events in the test cluster? Thanks. ---
**Author:** ![ginger](https://avatars.discourse-cdn.com/v4/letter/g/34f0e0/32.png) [@ginger](https://discuss.elastic.co/u/ginger)\ **Post date:** [February 9, 2018, 3:11pm UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/26 "2018-02-09T15:11:06Z")
> [@ywelsch](#): > > In case where this happens again in production, can you take stack dumps / heap dumps of the master? We will share the heap dump of the master next time. > [@ywelsch](#): > > Can you also provide more information on all the steps you took to reproduce this, in particular the ones which led to the 106945 shard-failed events in the test cluster? Thanks. Our new test cluster enviroment: **Elasticsearch cluster:** 3 machines each with 4 data nodes: 12 Cores_4, 64GB_4 memory Choose another machine as dedicated master Allocation awareness: cluster.routing.allocation.awareness.attributes: ip **Elasticsearch plugin:** Reject bulk request and shard bulk request when high JVM old generation usage large then 85%. **Steps to reproduce:** We use tc cmd to simulate the hardware failure and reproduce the problem: - Start the cluster - Create an index with 12 shards, each with 1 replica - Do heavy index on data nodes: multiple bulks concurrent on all data nodes continuely, each with 5k document, bulks cost 50% CPU of all data machine. - Use tc cmd to randomly drop packet of one data machine: ```auto tc qdisc add dev eth0 root netem loss 50% ``` ---
**Author:** ![ginger](https://avatars.discourse-cdn.com/v4/letter/g/34f0e0/32.png) [@ginger](https://discuss.elastic.co/u/ginger)\ **Post date:** [February 9, 2018, 3:13pm UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/27 "2018-02-09T15:13:40Z")
The code seems that it doesn't deduplicate shard failed tasks. It would be nice if you could provide a patch that deduplicate those task, so that we could check the theory right or not. ---
**Author:** ![ginger](https://avatars.discourse-cdn.com/v4/letter/g/34f0e0/32.png) [@ginger](https://discuss.elastic.co/u/ginger)\ **Post date:** [February 11, 2018, 5:32am UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/28 "2018-02-11T05:32:01Z")
@ywelsch We find that checking for dups are based on task identity in TaskBatcher.java, which will lead to too many duplicated shard failed tasks in \_cat/pending\_tasks. After changing IdentityHashMap to ConcurrentHashMap and overwrite equals method of ShardEntry, the shard failed tasks number drops sharply. Now the test cluster can recover to green. If checking for dups based on task identity is required for some cluster state update tasks? if it's just a bug? 😄 ---
**Author:** ![ywelsch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ywelsch/32/7751_2.png) [@ywelsch](https://discuss.elastic.co/u/ywelsch)\ **Post date:** [February 12, 2018, 8:18am UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/29 "2018-02-12T08:18:43Z")
Task identity is currently used to establish the relation between requests and responses (for each request there's exactly one response). Deduplicating the shard failed events can and should be done on the sender side (ShardStateAction), which is something that has been on my TODO list already. This will not only reduce the load on the master, but also reduce the amount of messages that have to flow between data and master node. I also want to check if there's anything that can be done to make TaskBatcher more efficient when dealing with a large number of tasks. I'll be off for 2 days now, but will open an issue and start working on a fix by the end of the week. ---
**Author:** ![ginger](https://avatars.discourse-cdn.com/v4/letter/g/34f0e0/32.png) [@ginger](https://discuss.elastic.co/u/ginger)\ **Post date:** [February 12, 2018, 9:13am UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/30 "2018-02-12T09:13:10Z")
Thanks very much. Please send us the issue link after you open it, and we will test after you fix it. ---
**Author:** ![ywelsch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ywelsch/32/7751_2.png) [@ywelsch](https://discuss.elastic.co/u/ywelsch)\ **Post date:** [February 15, 2018, 10:22am UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/31 "2018-02-15T10:22:32Z")
There's already an open issue that covers this: > We'll start working on a fix. ---
**Author:** ![ginger](https://avatars.discourse-cdn.com/v4/letter/g/34f0e0/32.png) [@ginger](https://discuss.elastic.co/u/ginger)\ **Post date:** [February 20, 2018, 2:24am UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/32 "2018-02-20T02:24:49Z")
Thanks. We subscribe this issue to wait for patch. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 20, 2018, 2:24am UTC](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658/33 "2018-03-20T02:24:55Z")
Closed on March 20, 2018, 2:24am UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/half-dead-node-lead-to-cluster-hang/113658.md?page=1)