# Write queue continue to rise
**URL:** https://discuss.elastic.co/t/write-queue-continue-to-rise/213652
**Category:** Elasticsearch
**Created:** [January 3, 2020, 5:28am UTC](https://discuss.elastic.co/t/write-queue-continue-to-rise/213652 "2020-01-03T05:28:14Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@ITzhangqiang](https://discuss.elastic.co/u/ITzhangqiang)
#### Post date: [January 3, 2020, 5:28am UTC](https://discuss.elastic.co/t/write-queue-continue-to-rise/213652/1 "2020-01-03T05:28:15Z")
One es node write queue continue to rise from about 08:00。But I set thread\_pool.write.queue\_size: 2000，why this node's write queue more than 2000?
And why queue Sudden rise?I check bulk request not much changed as usual。
---
### Author:  [@HenningAndersen](https://discuss.elastic.co/u/HenningAndersen)
#### Post date: [January 3, 2020, 11:51am UTC](https://discuss.elastic.co/t/write-queue-continue-to-rise/213652/2 "2020-01-03T11:51:10Z")
Hi @ITzhangqiang,
first of all, which version of Elasticsearch are you running?
It looks like all indexing halted and is queueing up instead, might be a deadlock. Seeing a jstack output from the node as well as the output of \_nodes/stats, \_nodes/hot\_threads and \_tasks should help reveal the cause.
---
### Author:  [@Armin\_Braun](https://discuss.elastic.co/u/Armin_Braun)
#### Post date: [January 3, 2020, 12:17pm UTC](https://discuss.elastic.co/t/write-queue-continue-to-rise/213652/3 "2020-01-03T12:17:14Z")
Linking your other thread here @ITzhangqiang
> [@Index writer memory Continue to rise](https://discuss.elastic.co/t/index-writer-memory-continue-to-rise/213514):
>
> hi，all： I have a question that has puzzled me for a long time。 One node in the cluster has too much index writer memory (other node is fline), and it keeps going up,and it lead to index throttling . And then I adjusted index.refresh.interval from 30s to 10s , but the situation has not improved much。 I also found this issued node's refresh queue has a large back-up， [image] why index writer cost so much memory?I found nothing in log As index writer memory continues to rise，in t…
I think these two things are connected. As I said there and @HenningAndersen here, we'll need the full `jstack` output to track down the dead-lock.
---
### Author:  [@ITzhangqiang](https://discuss.elastic.co/u/ITzhangqiang)
#### Post date: [January 3, 2020, 3:28pm UTC](https://discuss.elastic.co/t/write-queue-continue-to-rise/213652/4 "2020-01-03T15:28:06Z")
Thanks for your reply,
Finally,I upload related documents on github，url：
> **[ITzhangqiang/test\_Elasticsearch7.4](https://github.com/ITzhangqiang/test_Elasticsearch7.4)**
>
> some problem about es7.4. Contribute to ITzhangqiang/test\_Elasticsearch7.4 development by creating an account on GitHub.
@Armin_Braun
@HenningAndersen
---
### Author:  [@HenningAndersen](https://discuss.elastic.co/u/HenningAndersen)
#### Post date: [January 3, 2020, 4:40pm UTC](https://discuss.elastic.co/t/write-queue-continue-to-rise/213652/5 "2020-01-03T16:40:42Z")
@ITzhangqiang,
did the queuing of write requests resolve itself? I do not see any write requests queuing up in the `_nodes/stats` output. Which node had the issue?
---
### Author:  [@nhat](https://discuss.elastic.co/u/nhat)
#### Post date: [January 3, 2020, 5:23pm UTC](https://discuss.elastic.co/t/write-queue-continue-to-rise/213652/6 "2020-01-03T17:23:27Z")
@ITzhangqiang Can you please turn on and share the trace logs of IndexWriter?
```auto
PUT /_cluster/settings
{
"transient": {
"org.elasticsearch.index.engine.Engine.BD": "trace"
}
}
```
---
### Author:  [@ITzhangqiang](https://discuss.elastic.co/u/ITzhangqiang)
#### Post date: [January 4, 2020, 12:58am UTC](https://discuss.elastic.co/t/write-queue-continue-to-rise/213652/7 "2020-01-04T00:58:45Z")
data-02 had the issue"write queue rise".At that time ,I check the data-02 jstack file and jmap dump file ,did't find anything unusual. So I have to restart the data-02，after restart，the data-02 write queue is down.
I kept the data-02 jstack file in the issued time and upload it on git now,you can have a look.
Thanks
---
### Author:  [@nhat](https://discuss.elastic.co/u/nhat)
#### Post date: [January 4, 2020, 2:35am UTC](https://discuss.elastic.co/t/write-queue-continue-to-rise/213652/8 "2020-01-04T02:35:32Z")
@ITzhangqiang Not sure if you missed my message above. It would great if you can turn on and share trace logs from IndexWriter of `data-02`. Thank you!
---
### Author:  [@ITzhangqiang](https://discuss.elastic.co/u/ITzhangqiang)
#### Post date: [January 4, 2020, 4:07am UTC](https://discuss.elastic.co/t/write-queue-continue-to-rise/213652/9 "2020-01-04T04:07:28Z")
sorry,I just see your message,but data-02 is restarted,write queue is down.
I turn on the trace and upload logs .
path: test\_Elasticsearch7.4/blob/master/logs-from-es-data-in-es-data-2.txt
@nhat
---
### Author:  [@ITzhangqiang](https://discuss.elastic.co/u/ITzhangqiang)
#### Post date: [January 4, 2020, 4:54am UTC](https://discuss.elastic.co/t/write-queue-continue-to-rise/213652/10 "2020-01-04T04:54:46Z")
Another problem：data-06 refresh queue is abnorma from yesterday ，and now data-06 has been removed and added again and again by master（seems data-06 is hung，and can't response master's publication of cluster state in time ）,I don't konw how to break this situation except by rebooting.
>
---
### Author:  [@nhat](https://discuss.elastic.co/u/nhat)
#### Post date: [January 4, 2020, 5:26am UTC](https://discuss.elastic.co/t/write-queue-continue-to-rise/213652/11 "2020-01-04T05:26:37Z")
Thanks @ITzhangqiang. Sadly, the problematic node was restarted. Can you turn off the trace log and turn it on again when the problem reoccurs?
```auto
PUT /_cluster/settings
{
"transient": {
"org.elasticsearch.index.engine.Engine.BD": null
}
}
```
---
### Author:  [@nhat](https://discuss.elastic.co/u/nhat)
#### Post date: [January 4, 2020, 5:28am UTC](https://discuss.elastic.co/t/write-queue-continue-to-rise/213652/12 "2020-01-04T05:28:00Z")
@ITzhangqiang Can you also share ES log from `data-06`? Thank you!
---
### Author:  [@ITzhangqiang](https://discuss.elastic.co/u/ITzhangqiang)
#### Post date: [January 4, 2020, 5:47am UTC](https://discuss.elastic.co/t/write-queue-continue-to-rise/213652/13 "2020-01-04T05:47:35Z")
@nhat Thankyou for your reply，I upload data-06 log .
[https://github.com/ITzhangqiang/test\_Elasticsearch7.4/blob/master/logs-from-es-data-in-es-data-6%20(1).rar](https://github.com/ITzhangqiang/test_Elasticsearch7.4/blob/master/logs-from-es-data-in-es-data-6%20(1).rar)
---
### Author:  [@ITzhangqiang](https://discuss.elastic.co/u/ITzhangqiang)
#### Post date: [January 4, 2020, 5:49am UTC](https://discuss.elastic.co/t/write-queue-continue-to-rise/213652/14 "2020-01-04T05:49:20Z")
no problem.
---
### Author:  [@Armin\_Braun](https://discuss.elastic.co/u/Armin_Braun)
#### Post date: [January 5, 2020, 2:29pm UTC](https://discuss.elastic.co/t/write-queue-continue-to-rise/213652/15 "2020-01-05T14:29:31Z")
> [@ITzhangqiang](#):
>
> seems data-06 is hung，and can't response master's publication of cluster state in time
I wonder since all these problems appear to turn up on data-6 and the stack traces look like data-6 is simply taking forever on one thread that is `Runnable` doing work while some other thread is blocked on the lock held by the working thread, could it be that this is rather a system problem
with data node 6 that shows up via ES here?
It's somewhat suspicious that first a write thread gets seemingly stuck on doing work and now the CS applier state seems to have locked up.
@ITzhangqiang can you share some details on how data-06 is deployed? Is it running on some virtualization that is doing something along the lines of throttling CPU or overcommitting CPU across multiple VMs or so?
---
### Author:  [@ITzhangqiang](https://discuss.elastic.co/u/ITzhangqiang)
#### Post date: [January 5, 2020, 3:02pm UTC](https://discuss.elastic.co/t/write-queue-continue-to-rise/213652/16 "2020-01-05T15:02:38Z")
@Armin_Braun
Thankyou for your reply
My es7.4 cluster deployed on Kubernetes。One es instance deployed on one real physical machines。
By the way，today I restart some issued node，and cluster becomed green。But after several hours，problems again happen，look：
I will upload other logs on git [https://github.com/ITzhangqiang/test\_Elasticsearch7.4/tree/master/2020-01-05](https://github.com/ITzhangqiang/test_Elasticsearch7.4/tree/master/2020-01-05)
---
### Author:  [@ITzhangqiang](https://discuss.elastic.co/u/ITzhangqiang)
#### Post date: [January 6, 2020, 2:10am UTC](https://discuss.elastic.co/t/write-queue-continue-to-rise/213652/17 "2020-01-06T02:10:14Z")
@nhat ，hi，problem reoccurs，you can look es-data-6 log （ "logger.org.elasticsearch.index.engine.Engine.BD": "trace"） on
>
---
### Author:  [@nhat](https://discuss.elastic.co/u/nhat)
#### Post date: [January 6, 2020, 2:50am UTC](https://discuss.elastic.co/t/write-queue-continue-to-rise/213652/18 "2020-01-06T02:50:20Z")
Thanks @ITzhangqiang. I am looking at the log.
---
### Author:  [@ITzhangqiang](https://discuss.elastic.co/u/ITzhangqiang)
#### Post date: [January 6, 2020, 9:56am UTC](https://discuss.elastic.co/t/write-queue-continue-to-rise/213652/19 "2020-01-06T09:56:33Z")
hi,is there any new development on this issue?
I have to recover my cluster,so maybe I must restart some node. Do you need any other information about those issued nodes before I restart. I'd be happy to supply it.
Thanks!
@nhat
---
### Author:  [@nhat](https://discuss.elastic.co/u/nhat)
#### Post date: [January 6, 2020, 1:43pm UTC](https://discuss.elastic.co/t/write-queue-continue-to-rise/213652/20 "2020-01-06T13:43:07Z")
Hi @ITzhangqiang, not yet! Can you post the output of `GET /_stats?level=shards`, `GET /_tasks` and logs from the problematic nodes. Thank you.
[Next page](https://discuss.elastic.co/t/write-queue-continue-to-rise/213652.md?page=2)