# Get tasks hang and accumulate on 5.4.2
**URL:** https://discuss.elastic.co/t/get-tasks-hang-and-accumulate-on-5-4-2/92368
**Category:** Elasticsearch
**Created:** [July 8, 2017, 4:33pm UTC](https://discuss.elastic.co/t/get-tasks-hang-and-accumulate-on-5-4-2/92368 "2017-07-08T16:33:57Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@Tamas\_Pakolicz](https://discuss.elastic.co/u/Tamas_Pakolicz)
#### Post date: [July 8, 2017, 4:33pm UTC](https://discuss.elastic.co/t/get-tasks-hang-and-accumulate-on-5-4-2/92368/1 "2017-07-08T16:33:58Z")
Hi!
I have a problem with our Elasticsearch 5.4.2 cluster.
We have 52 nodes of which 40 are data nodes. On our non-master data nodes tasks are accumulating and get stuck. A full cluster restart did not solve our problem, the accumulating tasks came back quickly. If we restart a node which we have a problem, it just transfers to another nodes like a disease.
The detailed task list shows entries like this one:
```
"node" : "Q_pZKTu4R9-wldTlrPsLcA",
"id" : 29589541,
"type" : "netty",
"action" : "indices:data/read/get",
"description" : "",
"start_time_in_millis" : 1499508299639,
"running_time_in_nanos" : 16858865861967,
"cancellable" : false
},
```
The problem first surfaced after we upgraded the cluster to 5.4. We have never had it before.
---
### Author:  [@thiago](https://discuss.elastic.co/u/thiago)
#### Post date: [July 8, 2017, 9:18pm UTC](https://discuss.elastic.co/t/get-tasks-hang-and-accumulate-on-5-4-2/92368/2 "2017-07-08T21:18:42Z")
Does your application updates a document before it tries to retrieve it with the Get API?
---
### Author:  [@Tamas\_Pakolicz](https://discuss.elastic.co/u/Tamas_Pakolicz)
#### Post date: [July 8, 2017, 9:27pm UTC](https://discuss.elastic.co/t/get-tasks-hang-and-accumulate-on-5-4-2/92368/3 "2017-07-08T21:27:35Z")
No, it's just a get after a search.
---
### Author:  [@thiago](https://discuss.elastic.co/u/thiago)
#### Post date: [July 8, 2017, 9:46pm UTC](https://discuss.elastic.co/t/get-tasks-hang-and-accumulate-on-5-4-2/92368/4 "2017-07-08T21:46:22Z")
I've asked this because the realtime get has changed recently. It will do a refresh before getting the document if it changed.
But since you are not updating it, then it is not clear to me why the task list is filling with get action.
Maybe @jasontedor knows more?
---
### Author:  [@jasontedor](https://discuss.elastic.co/u/jasontedor)
#### Post date: [July 9, 2017, 2:35am UTC](https://discuss.elastic.co/t/get-tasks-hang-and-accumulate-on-5-4-2/92368/5 "2017-07-09T02:35:17Z")
I think that we need to see output from the hot threads API and also the output of `/_nodes/stats?filter_path=**.thread_pool.get`.
---
### Author:  [@Tamas\_Pakolicz](https://discuss.elastic.co/u/Tamas_Pakolicz)
#### Post date: [July 9, 2017, 4:23am UTC](https://discuss.elastic.co/t/get-tasks-hang-and-accumulate-on-5-4-2/92368/7 "2017-07-09T04:23:11Z")
The cluster was completely restarted ~7 hours ago, so there are not too many hung tasks (~17k at this moment) but the application hangs after a couple minutes with ES read timeouts and nothing can be seen in the ES's logs. And the app did not change.
Hot threads:
[https://drive.google.com/open?id=0B7\_2U7nvLS3TcjRfeDdaeVZyYnc](https://drive.google.com/open?id=0B7_2U7nvLS3TcjRfeDdaeVZyYnc)
thread.pool.get:
[https://drive.google.com/open?id=0B7\_2U7nvLS3TbVdxY0FaOWgxTm8](https://drive.google.com/open?id=0B7_2U7nvLS3TbVdxY0FaOWgxTm8)
---
### Author:  [@Attila\_Nagy](https://discuss.elastic.co/u/Attila_Nagy)
#### Post date: [July 9, 2017, 7:37am UTC](https://discuss.elastic.co/t/get-tasks-hang-and-accumulate-on-5-4-2/92368/8 "2017-07-09T07:37:57Z")
BTW, for search and other operations task api shows which index is used. Get doesn't. Is it intentional?
Maybe it's relevant which index has the problem (if it can be narrowed to one).
---
### Author:  [@Tamas\_Pakolicz](https://discuss.elastic.co/u/Tamas_Pakolicz)
#### Post date: [July 10, 2017, 6:27am UTC](https://discuss.elastic.co/t/get-tasks-hang-and-accumulate-on-5-4-2/92368/9 "2017-07-10T06:27:33Z")
After a really complete restart (master nodes too), the accumulation of tasks seems to have stopped. Anyway, I would really be a lot calmer if I had known what caused this phenomenon.
---
### Author:  [@Attila\_Nagy](https://discuss.elastic.co/u/Attila_Nagy)
#### Post date: [July 24, 2017, 2:05pm UTC](https://discuss.elastic.co/t/get-tasks-hang-and-accumulate-on-5-4-2/92368/10 "2017-07-24T14:05:34Z")
I've opened an issue from this:
>
because we still don't know what happened and how this could be solved (apart from getting every possible load off from elasticsearch and several cluster restarts).
Please tell me if we can provide more info to help to sort this out.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 21, 2017, 2:05pm UTC](https://discuss.elastic.co/t/get-tasks-hang-and-accumulate-on-5-4-2/92368/11 "2017-08-21T14:05:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.