### Author:  [@Armin\_Braun](https://discuss.elastic.co/u/Armin_Braun)
#### Post date: [January 2, 2020, 7:31am UTC](https://discuss.elastic.co/t/index-writer-memory-continue-to-rise/213514/2 "2020-01-02T07:31:50Z")
Hi @ITzhangqiang
Are you triggering refreshes manually at a high rate somehow? (it certainly looks like it).
Maybe those refreshes only trigger shards residing on `es-data-8` causing it to become overloaded and unable to flush its index writers.
Can you check the (tasks API)[[https://www.elastic.co/guide/en/elasticsearch/reference/current/tasks.html#tasks-api-examples](https://www.elastic.co/guide/en/elasticsearch/reference/current/tasks.html#tasks-api-examples)] for `es-data-8` and share the results? That should help figure out where those refreshes are coming from I think.
---
### Author:  [@ITzhangqiang](https://discuss.elastic.co/u/ITzhangqiang)
#### Post date: [January 2, 2020, 8:07am UTC](https://discuss.elastic.co/t/index-writer-memory-continue-to-rise/213514/3 "2020-01-02T08:07:50Z")
> [@Armin\_Braun](#):
>
> That should help figure out where those refreshes are coming from I think.
Thank you for your reply，
I don't refresh index manually,I use logstash transport data from kafka to elasticsearch.
and I use logstash double writer data to es6.3 cluster and es7.4 cluster, es6.3 cluster never happend this problem。
Because of es-data-8 node's memory reached 95% of heap size(lead to CircuitBreakingException )，so task API can't return correct result .(Execption：[circuit\_breaking\_exception] [parent] Data too large, data for [\] would be [32273530784/30gb], which is larger than the limit of [31621696716/29.4gb])
I use task API try many time，return bellow result（i think it is incomplete，previous it can return many refresh task ）
Another information：I check shard on data8, most index shard is evenly distributed to all nodes，so index refresh will happen in all node
---
### Author:  [@Armin\_Braun](https://discuss.elastic.co/u/Armin_Braun)
#### Post date: [January 2, 2020, 11:22am UTC](https://discuss.elastic.co/t/index-writer-memory-continue-to-rise/213514/4 "2020-01-02T11:22:56Z")
Thanks for the tasks list @ITzhangqiang
it looks like the refresh is stuck/dead-locked somehow (it's running for 16h+ already!). That seems to be the problem. Can you take a thread-dump on `data8`, so we can start tracking down why/where it dead-locked maybe?
Thanks!
---
### Author:  [@ITzhangqiang](https://discuss.elastic.co/u/ITzhangqiang)
#### Post date: [January 2, 2020, 11:46am UTC](https://discuss.elastic.co/t/index-writer-memory-continue-to-rise/213514/5 "2020-01-02T11:46:04Z")
unfortunately, I deployed the cluster on Kubernetes，in docker container，i cat run jstack。 😭
Do you know how to set it up
---
### Author:  [@Armin\_Braun](https://discuss.elastic.co/u/Armin_Braun)
#### Post date: [January 2, 2020, 11:53am UTC](https://discuss.elastic.co/t/index-writer-memory-continue-to-rise/213514/6 "2020-01-02T11:53:39Z")
@ITzhangqiang
it should work fine using `nsenter` into the Docker container. The easiest way of doing that, that I know of is [https://github.com/jpetazzo/nsenter](https://github.com/jpetazzo/nsenter) (unless you have `nsenter` properly working already and know how to do it :)) That should allow you to `jstack` just fine without permission issues.
---
### Author:  [@ITzhangqiang](https://discuss.elastic.co/u/ITzhangqiang)
#### Post date: [January 2, 2020, 12:12pm UTC](https://discuss.elastic.co/t/index-writer-memory-continue-to-rise/213514/7 "2020-01-02T12:12:51Z")
thank you very much,I already restart es-data-8,and cluster's health is recoverd green.so thread dump can't tracking the issue. but i think this problem will happen again. if happen ,I will notice you.
Thank you once again!
---
### Author:  [@ITzhangqiang](https://discuss.elastic.co/u/ITzhangqiang)
#### Post date: [January 3, 2020, 8:54am UTC](https://discuss.elastic.co/t/index-writer-memory-continue-to-rise/213514/8 "2020-01-03T08:54:53Z")
Hi， another node happen this problem
I check the jstack file ，found：locked \<0x000000128539b510\>
@Armin_Braun
---
### Author:  [@Armin\_Braun](https://discuss.elastic.co/u/Armin_Braun)
#### Post date: [January 3, 2020, 9:41am UTC](https://discuss.elastic.co/t/index-writer-memory-continue-to-rise/213514/9 "2020-01-03T09:41:17Z")
@ITzhangqiang can you provide the full jstack file here so I can take a look?
---
### Author:  [@ITzhangqiang](https://discuss.elastic.co/u/ITzhangqiang)
#### Post date: [January 3, 2020, 9:59am UTC](https://discuss.elastic.co/t/index-writer-memory-continue-to-rise/213514/10 "2020-01-03T09:59:05Z")
Thankyou for your reply：
I run jstack twice，all found same blocked thread。
I found many node happen this problem。
![]()
在 2020-01-03 17:51:23，"Armin Braun via Discuss the Elastic Stack" [elastic@discoursemail.com](mailto:elastic@discoursemail.com) 写道：
(Attachment jstack06\_2 is missing)
(Attachment jstack06 is missing)
---
### Author:  [@ITzhangqiang](https://discuss.elastic.co/u/ITzhangqiang)
#### Post date: [January 3, 2020, 10:02am UTC](https://discuss.elastic.co/t/index-writer-memory-continue-to-rise/213514/11 "2020-01-03T10:02:39Z")
![]()
I receive this info ，can't upload file
在 2020-01-03 17:51:23，"Armin Braun via Discuss the Elastic Stack" [elastic@discoursemail.com](mailto:elastic@discoursemail.com) 写道：
---
### Author:  [@Armin\_Braun](https://discuss.elastic.co/u/Armin_Braun)
#### Post date: [January 3, 2020, 10:08am UTC](https://discuss.elastic.co/t/index-writer-memory-continue-to-rise/213514/12 "2020-01-03T10:08:55Z")
@ITzhangqiang
can you upload the jstack files to e.g. [https://pastebin.com](https://pastebin.com) maybe and link them here? That should work fine 🙂
---
### Author:  [@nhat](https://discuss.elastic.co/u/nhat)
#### Post date: [January 3, 2020, 5:41pm UTC](https://discuss.elastic.co/t/index-writer-memory-continue-to-rise/213514/13 "2020-01-03T17:41:24Z")
I believe those refreshes are queued up by IndexingMemoryController. I will take a closer look.
---
### Author:  [@ITzhangqiang](https://discuss.elastic.co/u/ITzhangqiang)
#### Post date: [January 9, 2020, 6:36am UTC](https://discuss.elastic.co/t/index-writer-memory-continue-to-rise/213514/14 "2020-01-09T06:36:03Z")
hi,
@nhat
@Armin_Braun
I'm sorry to bother you, That's still the question that elastic search 7.4 always have some node which index writer hold too many memory.I believe this is a big issue. （es 6.3 never find this issue for a year）Help me to find what the reason behind！We must solve this problem！
---
### Author:  [@nhat](https://discuss.elastic.co/u/nhat)
#### Post date: [January 9, 2020, 1:57pm UTC](https://discuss.elastic.co/t/index-writer-memory-continue-to-rise/213514/15 "2020-01-09T13:57:09Z")
@ITzhangqiang No worries, we are here to help. Can you share the shard-level stats of your cluster (`GET /_stats?level=shards`)?
---
### Author:  [@nhat](https://discuss.elastic.co/u/nhat)
#### Post date: [January 10, 2020, 2:35am UTC](https://discuss.elastic.co/t/index-writer-memory-continue-to-rise/213514/16 "2020-01-10T02:35:48Z")
@ITzhangqiang I have merged [https://github.com/elastic/elasticsearch/pull/50769](https://github.com/elastic/elasticsearch/pull/50769), which should avoid flooding the refresh thread pool.
---
### Author:  [@ITzhangqiang](https://discuss.elastic.co/u/ITzhangqiang)
#### Post date: [January 10, 2020, 5:30am UTC](https://discuss.elastic.co/t/index-writer-memory-continue-to-rise/213514/17 "2020-01-10T05:30:05Z")
Thank you very much,
@nhat
I found index writer's abnormal mabye related to some special index.
see the index below：
This index cause data-09 node's writer memory abnormal，see data-09 node below：
I check shard on this index ,found one shard hold a big index writer memory:
I upload data-09 jstack file and this abnormal index's shard detail state
> **[ITzhangqiang/test\_Elasticsearch7.4](https://github.com/ITzhangqiang/test_Elasticsearch7.4/tree/master/2020-01-10)**
>
> some problem about es7.4. Contribute to ITzhangqiang/test\_Elasticsearch7.4 development by creating an account on GitHub.
---
### Author:  [@nhat](https://discuss.elastic.co/u/nhat)
#### Post date: [January 11, 2020, 4:49pm UTC](https://discuss.elastic.co/t/index-writer-memory-continue-to-rise/213514/18 "2020-01-11T16:49:35Z")
Hi @ITzhangqiang,
> "uncommitted\_size\_in\_bytes" : 12605908804
The uncommitted translog should not go above 512MB per shard by default. Did you change any translog setting? Can you share the logs from the node data-09?
One theory that I have is that the throttling does not work well. Can you add
`-Des.index.memory.max_index_buffer_size=256mb` to `config/jvm.options` on some nodes then restart them. Please let me know if the problem goes away on those nodes. Thank you.
---
### Author:  [@ITzhangqiang](https://discuss.elastic.co/u/ITzhangqiang)
#### Post date: [January 13, 2020, 2:43am UTC](https://discuss.elastic.co/t/index-writer-memory-continue-to-rise/213514/19 "2020-01-13T02:43:17Z")
Hi @nhat
Most of index ，I changed translog settings：
"translog" : {
"sync\_interval" : "60s",
"durability" : "async",
"flush\_threshold\_size": "1gb"
}
I think I have found the problem.I checked the abnormal index carefully. Found I manually specify
the "document\_id" field,but index data is problematic frequently,lead to millions of "document\_id" are same ,so documents keep updating .
(as I see in the jstack file ,most of write thread is waiting “doc\_id lock” ：
`at org.elasticsearch.index.engine.LiveVersionMap.acquireLock(LiveVersionMap.java:473) at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:856)`) .
It's the problematic index that lead to those issue："index writer memory rise","write queue rise".
After I change the problematic index ,everything is back to normal.
Thank you very much for your help during this period.
Thanks again!
---
### Author:  [@nhat](https://discuss.elastic.co/u/nhat)
#### Post date: [January 13, 2020, 1:43pm UTC](https://discuss.elastic.co/t/index-writer-memory-continue-to-rise/213514/20 "2020-01-13T13:43:38Z")
@ITzhangqiang Glad to hear that the problem was solved. You're welcome and thank you for collaborating.
[Next page](https://discuss.elastic.co/t/index-writer-memory-continue-to-rise/213514.md?page=2)