# Java Heap dump consuming disk space **URL:** \ **Category:** Elasticsearch\ **Created:** [September 1, 2021, 3:36pm UTC](https://discuss.elastic.co/t/java-heap-dump-consuming-disk-space/283068 "2021-09-01T15:36:00Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![krish0608](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krish0608/32/6797_2.png) [@krish0608](https://discuss.elastic.co/u/krish0608)\ **Post date:** [September 1, 2021, 3:36pm UTC](https://discuss.elastic.co/t/java-heap-dump-consuming-disk-space/283068/1 "2021-09-01T15:36:00Z")
We have been getting java heap out of memory errors quite regularly, so this time we decided to take the heap dump and analyze it. But, unfortunately its tough for us to make anything out of it. So we are seeking for some help. Also in jvm.options file we have disabled below mentioned parameters:- #-XX:+UseConcMarkSweepGC #-XX:CMSInitiatingOccupancyFraction=75 #-XX:+UseCMSInitiatingOccupancyOnly Just curious to know is these parameter results in creation of Java heap dump files. We are using 7.9.3 version and also just for info we are using 17 Nodes clusters. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 1, 2021, 10:03pm UTC](https://discuss.elastic.co/t/java-heap-dump-consuming-disk-space/283068/2 "2021-09-01T22:03:19Z")
What is the output from the `_cluster/stats?pretty&human` API? ---
**Author:** ![krish0608](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krish0608/32/6797_2.png) [@krish0608](https://discuss.elastic.co/u/krish0608)\ **Post date:** [September 2, 2021, 3:16am UTC](https://discuss.elastic.co/t/java-heap-dump-consuming-disk-space/283068/3 "2021-09-02T03:16:08Z")
@warkolm - Thanks for your revert. Also forget to mention about that we have commented above parameters because recently we have upgraded our cluster from 7.0.1 to 7.9.3 and after upgradation these 3 parameters causing issue and we are not able to restart of Elasticsearch service. Thats why we have commented it out. Here is output of API:- { "\_nodes" : { "total" : 4, "successful" : 4, "failed" : 0 }, "cluster\_name" : "XCS1\_LOG\_PRODUCTION", "cluster\_uuid" : "DxQkLc6IRFaaKyfsYMnaqw", "timestamp" : 1630552294934, "status" : "green", "indices" : { "count" : 50, "shards" : { "total" : 100, "primaries" : 50, "replication" : 1.0, "index" : { "shards" : { "min" : 2, "max" : 2, "avg" : 2.0 }, "primaries" : { "min" : 1, "max" : 1, "avg" : 1.0 }, "replication" : { "min" : 1.0, "max" : 1.0, "avg" : 1.0 } } }, "docs" : { "count" : 38690, "deleted" : 1987 }, "store" : { "size\_in\_bytes" : 2207040041, "reserved\_in\_bytes" : 0 }, "fielddata" : { "memory\_size\_in\_bytes" : 0, "evictions" : 0 }, "query\_cache" : { "memory\_size\_in\_bytes" : 118000, "total\_count" : 1445, "hit\_count" : 129, "miss\_count" : 1316, "cache\_size" : 65, "cache\_count" : 65, "evictions" : 0 }, "completion" : { "size\_in\_bytes" : 0 }, "segments" : { "count" : 361, "memory\_in\_bytes" : 1456984, "terms\_memory\_in\_bytes" : 1068528, "stored\_fields\_memory\_in\_bytes" : 169816, "term\_vectors\_memory\_in\_bytes" : 0, "norms\_memory\_in\_bytes" : 68544, "points\_memory\_in\_bytes" : 0, "doc\_values\_memory\_in\_bytes" : 150096, "index\_writer\_memory\_in\_bytes" : 865280, "version\_map\_memory\_in\_bytes" : 0, "fixed\_bit\_set\_memory\_in\_bytes" : 8696, "max\_unsafe\_auto\_id\_timestamp" : 1630546435647, "file\_sizes" : { } }, "mappings" : { "field\_types" : [ { "name" : "alias", "count" : 1, "index\_count" : 1 }, { "name" : "binary", "count" : 9, "index\_count" : 1 }, { "name" : "boolean", "count" : 106, "index\_count" : 40 }, { "name" : "byte", "count" : 33, "index\_count" : 33 }, { "name" : "date", "count" : 239, "index\_count" : 49 }, { "name" : "double", "count" : 1, "index\_count" : 1 }, { "name" : "flattened", "count" : 9, "index\_count" : 1 }, { "name" : "float", "count" : 3, "index\_count" : 1 }, { "name" : "geo\_point", "count" : 1, "index\_count" : 1 }, { "name" : "integer", "count" : 29, "index\_count" : 2 }, { "name" : "ip", "count" : 2, "index\_count" : 1 }, { "name" : "keyword", "count" : 817, "index\_count" : 49 }, { "name" : "long", "count" : 150, "index\_count" : 46 }, { "name" : "nested", "count" : 16, "index\_count" : 6 }, { "name" : "object", "count" : 374, "index\_count" : 50 }, { "name" : "short", "count" : 66, "index\_count" : 33 }, { "name" : "text", "count" : 319, "index\_count" : 48 } ] }, "analysis" : { "char\_filter\_types" : , "tokenizer\_types" : , "filter\_types" : , "analyzer\_types" : , "built\_in\_char\_filters" : , "built\_in\_tokenizers" : , "built\_in\_filters" : , "built\_in\_analyzers" : } }, "nodes" : { "count" : { "total" : 4, "coordinating\_only" : 0, "data" : 4, "ingest" : 4, "master" : 4, "ml" : 4, "remote\_cluster\_client" : 4, "transform" : 4, "voting\_only" : 0 }, "versions" : [ "7.9.3" ], "os" : { "available\_processors" : 16, "allocated\_processors" : 16, "names" : [ { "name" : "Linux", "count" : 4 } ], "pretty\_names" : [ { "pretty\_name" : "Red Hat Enterprise Linux Server 7.9 (Maipo)", "count" : 4 } ], "mem" : { "total\_in\_bytes" : 66627600384, "free\_in\_bytes" : 12966031360, "used\_in\_bytes" : 53661569024, "free\_percent" : 19, "used\_percent" : 81 } }, "process" : { "cpu" : { "percent" : 0 }, "open\_file\_descriptors" : { "min" : 486, "max" : 560, "avg" : 511 } }, "jvm" : { "max\_uptime\_in\_millis" : 84338505, "versions" : [ { "version" : "15", "vm\_name" : "OpenJDK 64-Bit Server VM", "vm\_version" : "15+36-1562", "vm\_vendor" : "Oracle Corporation", "bundled\_jdk" : true, "using\_bundled\_jdk" : true, "count" : 4 } ], "mem" : { "heap\_used\_in\_bytes" : 15199421360, "heap\_max\_in\_bytes" : 33319550976 }, "threads" : 274 }, "fs" : { "total\_in\_bytes" : 42907729920, "free\_in\_bytes" : 25755017216, "available\_in\_bytes" : 25755017216 }, "plugins" : , "network\_types" : { "transport\_types" : { "security4" : 4 }, "http\_types" : { "security4" : 4 } }, "discovery\_types" : { "zen" : 4 }, "packaging\_types" : [ { "flavor" : "default", "type" : "rpm", "count" : 4 } ], "ingest" : { "number\_of\_pipelines" : 4, "processor\_stats" : { "date" : { "count" : 0, "failed" : 0, "current" : 0, "time\_in\_millis" : 0 }, "gsub" : { "count" : 0, "failed" : 0, "current" : 0, "time\_in\_millis" : 0 }, "rename" : { "count" : 0, "failed" : 0, "current" : 0, "time\_in\_millis" : 0 }, "script" : { "count" : 0, "failed" : 0, "current" : 0, "time\_in\_millis" : 0 }, "set" : { "count" : 0, "failed" : 0, "current" : 0, "time\_in\_millis" : 0 } } } } } ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 2, 2021, 4:00am UTC](https://discuss.elastic.co/t/java-heap-dump-consuming-disk-space/283068/4 "2021-09-02T04:00:46Z")
Please format your code/logs/config using the `` button, or markdown style back ticks. It helps to make things easy to read which helps us help you 🙂 Is your heap size set to ~15GB per node? ---
**Author:** ![krish0608](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krish0608/32/6797_2.png) [@krish0608](https://discuss.elastic.co/u/krish0608)\ **Post date:** [September 2, 2021, 5:52am UTC](https://discuss.elastic.co/t/java-heap-dump-consuming-disk-space/283068/5 "2021-09-02T05:52:34Z")
Yes i guess. Is something wrong i have configured.? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 2, 2021, 5:53am UTC](https://discuss.elastic.co/t/java-heap-dump-consuming-disk-space/283068/6 "2021-09-02T05:53:03Z")
Just checking. What do your Elasticsearch logs show in terms of the OOM? ---
**Author:** ![krish0608](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krish0608/32/6797_2.png) [@krish0608](https://discuss.elastic.co/u/krish0608)\ **Post date:** [September 2, 2021, 6:04am UTC](https://discuss.elastic.co/t/java-heap-dump-consuming-disk-space/283068/7 "2021-09-02T06:04:30Z")
```auto [2021-08-30T22:56:36,561][INFO][o.e.i.b.HierarchyCircuitBreakerService] [do-loguip701.prod.mdgapp.net] GC did bring memory usage down, before [7834664800], after [4785620880], allocations [1], duration [82] [2021-08-30T23:21:43,040][INFO][o.e.c.s.ClusterApplierService] [do-loguip701.prod.mdgapp.net] removed {{do-loguip801.prod.mdgapp.net}{BTPvv7gHRtuFwaLYPHQWDA}{3ZBGGg94Rb2psre_Q0iegA}{10.125.129.41}{10.125.129.41:9300}{dilmrt}{ml.machine_memory=16656900096, ml.max_open_jobs=20, xpack.installed=true, logess=ui, transform.node=true}}, term: 811, version: 319691, reason: ApplyCommitRequest{term=811, version=319691, sourceNode={do-loguip702.prod.mdgapp.net}{JsxscRxGQtSnaeVXL_Hf8g}{qF_Kh0bIQj2drsxZ4JpP3Q}{10.124.129.8}{10.124.129.8:9300}{dilmrt}{ml.machine_memory=16656900096, ml.max_open_jobs=20, xpack.installed=true, logess=ui, transform.node=true}} Caused by: java.lang.IllegalArgumentException: Text fields are not optimised for operations that require per-document field data like aggre... [2021-08-30T17:36:49,398][INFO][o.e.i.b.HierarchyCircuitBreakerService] [do-loguip701.prod.mdgapp.net] attempting to trigger G1GC due to high heap usage [7913509344] [2021-08-30T17:41:50,401][INFO][o.e.i.b.HierarchyCircuitBreakerService] [do-loguip701.prod.mdgapp.net] attempting to trigger G1GC due to high heap usage [7941661376] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final] at java.lang.Thread.run(Thread.java:832) [?:?] Caused by: org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks at org.elasticsearch.tasks.TaskManager$CancellableTaskHolder.registerChildNode(TaskManager.java:522) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.tasks.TaskManager.registerChildNode(TaskManager.java:213) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:626) ~[elasticsearch-7.9.3.jar:7.9.3] ... 66 more [2021-08-31T22:47:52,610][INFO][o.e.t.TaskCancellationService] [do-loguip701.prod.mdgapp.net] failed to remove the parent ban for task OiXW6LsqQwqbTYnX8dr4QA:296342 on node {do-logwarmp801.prod.mdgapp.net}{v-eOjR5rQRqzD1bVuPSgrA}{C-BLLoRpQJS3TdX-g1yKUw}{10.125.129.53}{10.125.129.53:9300}{dlrt}{ml.machine_memory=18769842176, ml.max_open_jobs=20, xpack.installed=true, logess=warm, transform.node=true} [2021-08-31T22:47:52,872][WARN][o.e.t.TaskCancellationService] [do-loguip701.prod.mdgapp.net] Cannot send ban for tasks with the parent [OiXW6LsqQwqbTYnX8dr4QA:296682] to the node [{do-loghotp703.prod.mdgapp.net}{WxRnUg8XS_2ml1Ndwd9ENw}{hqHF-GKWRImltSr3JaqiLA}{10.124.129.58}{10.124.129.58:9300}{dilrt}{ml.machine_memory=33566654464, ml.max_open_jobs=20, xpack.installed=true, logess=hot, transform.node=true}] [2021-08-31T22:47:52,883][INFO][o.e.t.TaskCancellationService] [do-loguip701.prod.mdgapp.net] failed to remove the parent ban for task OiXW6LsqQwqbTYnX8dr4QA:296682 on node {do-loghotp703.prod.mdgapp.net}{WxRnUg8XS_2ml1Ndwd9ENw}{hqHF-GKWRImltSr3JaqiLA}{10.124.129.58}{10.124.129.58:9300}{dilrt}{ml.machine_memory=33566654464, ml.max_open_jobs=20, xpack.installed=true, logess=hot, transform.node=true} [2021-08-31T22:52:50,765][WARN][o.e.t.TaskCancellationService] [do-loguip701.prod.mdgapp.net] Cannot send ban for tasks with the parent [OiXW6LsqQwqbTYnX8dr4QA:312162] to the node [{do-loghotp701.prod.mdgapp.net}{j91jdhtGQUG9SWG4L05LHA}{ugHoG47aS_-UA4C3PtzviQ}{10.124.129.26}{10.124.129.26:9300}{dilrt}{ml.machine_memory=33566654464, ml.max_open_jobs=20, xpack.installed=true, logess=hot, transform.node=true}] [2021-08-31T22:52:50,781][INFO][o.e.t.TaskCancellationService] [do-loguip701.prod.mdgapp.net] failed to remove the parent ban for task OiXW6LsqQwqbTYnX8dr4QA:312162 on node {do-loghotp701.prod.mdgapp.net}{j91jdhtGQUG9SWG4L05LHA}{ugHoG47aS_-UA4C3PtzviQ}{10.124.129.26}{10.124.129.26:9300}{dilrt}{ml.machine_memory=33566654464, ml.max_open_jobs=20, xpack.installed=true, logess=hot, transform.node=true} [2021-08-31T22:52:51,373][WARN][o.e.t.TaskCancellationService] [do-loguip701.prod.mdgapp.net] Cannot send ban for tasks with the parent [OiXW6LsqQwqbTYnX8dr4QA:312268] to the node [{do-loghotp701.prod.mdgapp.net}{j91jdhtGQUG9SWG4L05LHA}{ugHoG47aS_-UA4C3PtzviQ}{10.124.129.26}{10.124.129.26:9300}{dilrt}{ml.machine_memory=33566654464, ml.max_open_jobs=20, xpack.installed=true, logess=hot, transform.node=true}] [2021-08-31T22:52:51,373][WARN][o.e.t.TaskCancellationService] [do-loguip701.prod.mdgapp.net] Cannot send ban for tasks with the parent [OiXW6LsqQwqbTYnX8dr4QA:312268] to the node [{do-logwarmp804.prod.mdgapp.net}{Iur-7puaTlyCg9omtKQ1SA}{Qj7eA5UORRGyFVrYsIhkuw}{10.125.129.22}{10.125.129.22:9300}{dlrt}{ml.machine_memory=18769842176, ml.max_open_jobs=20, xpack.installed=true, logess=warm, transform.node=true}] [2021-08-31T22:52:51,373][DEBUG][o.e.a.s.TransportSearchAction] [do-loguip701.prod.mdgapp.net] [logstash-requestjail-2021.08.31-000498][0], node[dtOI6jD2RCSubXL4wS1qrw], [P], s[STARTED], a[id=R3pdffNkTn6DWgVKje0zhg]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[*:logstash-requestjail*], indicesOptions=IndicesOptions[ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, expand_wildcards_hidden=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=false], types=[], routing='null', preference='1630450359304', requestCache=true, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=64, preFilterShardSize=1, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=false, source={"size":500,"query":{"bool":{"must":[{"query_string":{"query":"*","default_field":"*","fields":[],"type":"best_fields","default_operator":"or","max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"analyze_wildcard":true,"time_zone":"America/Denver","escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}},{"match_all":{"boost":1.0}}],"filter":[{"match_phrase":{"marketer":{"query":"TC","slop":0,"zero_terms_query":"NONE","boost":1.0}}},{"match_phrase":{"environment.keyword":{"query":"production","slop":0,"zero_terms_query":"NONE","boost":1.0}}},{"range":{"@timestamp":{"from":"2021-08-31T06:00:00.000Z","to":"2021-09-01T05:59:59.999Z","include_lower":true,"include_upper":true,"format":"strict_date_optional_time","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"version":true,"_source":{"includes":[],"excludes":[]},"stored_fields":"*","docvalue_fields":[{"field":"@timestamp","format":"date_time"}],"script_fields":{},"sort":[],"track_total_hits":2147483647}}] lastShard [true] org.elasticsearch.transport.TransportException: failure to send at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:659) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:703) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:695) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] ``` is this helpful for you? ---
**Author:** ![krish0608](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krish0608/32/6797_2.png) [@krish0608](https://discuss.elastic.co/u/krish0608)\ **Post date:** [September 2, 2021, 3:52pm UTC](https://discuss.elastic.co/t/java-heap-dump-consuming-disk-space/283068/8 "2021-09-02T15:52:17Z")
```auto Caused by: org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks at org.elasticsearch.tasks.TaskManager$CancellableTaskHolder.registerChildNode(TaskManager.java:522) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.tasks.TaskManager.registerChildNode(TaskManager.java:213) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:626) ~[elasticsearch-7.9.3.jar:7.9.3] ... 66 more [2021-09-02T15:11:33,797][INFO][o.e.m.j.JvmGcMonitorService] [do-loguip702.prod.mdgapp.net] [gc][127471] overhead, spent [278ms] collecting in the last [1s] [2021-09-02T15:24:17,501][INFO][o.e.i.b.HierarchyCircuitBreakerService] [do-loguip702.prod.mdgapp.net] attempting to trigger G1GC due to high heap usage [8109806080] [2021-09-02T15:24:17,526][INFO][o.e.i.b.HierarchyCircuitBreakerService] [do-loguip702.prod.mdgapp.net] GC did bring memory usage down, before [8109806080], after [5376480440], allocations [37], duration [26] [2021-09-02T15:24:19,452][WARN][o.e.m.j.JvmGcMonitorService] [do-loguip702.prod.mdgapp.net] [gc][128236] overhead, spent [515ms] collecting in the last [1s] [2021-09-02T15:24:20,453][INFO][o.e.m.j.JvmGcMonitorService] [do-loguip702.prod.mdgapp.net] [gc][128237] overhead, spent [439ms] collecting in the last [1s] [2021-09-02T15:24:24,859][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [do-loguip702.prod.mdgapp.net] fatal error in thread [elasticsearch[do-loguip702.prod.mdgapp.net][write][T#3]], exiting java.lang.OutOfMemoryError: Java heap space ``` Here is latest got i have just recieved. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 2, 2021, 10:20pm UTC](https://discuss.elastic.co/t/java-heap-dump-consuming-disk-space/283068/9 "2021-09-02T22:20:48Z")
Thanks. What's in hot threads on that node? ---
**Author:** ![krish0608](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krish0608/32/6797_2.png) [@krish0608](https://discuss.elastic.co/u/krish0608)\ **Post date:** [September 3, 2021, 9:02am UTC](https://discuss.elastic.co/t/java-heap-dump-consuming-disk-space/283068/10 "2021-09-03T09:02:08Z")
@warkolm -I think you are lookint for \_nodes/hot\_threads output. Please correct if am wrong. ```auto ::: {do-loguip801.prod.mdgapp.net}{BTPvv7gHRtuFwaLYPHQWDA}{Zq2FUExlTByB8Y_dYVPqxA}{10.125.129.41}{10.125.129.41:9300}{dilmrt}{ml.machine_memory=16656900096, ml.max_open_jobs=20, xpack.installed=true, logess=ui, transform.node=true} Hot threads at 2021-09-03T08:58:37.150Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true: ::: {do-loguip702.prod.mdgapp.net}{JsxscRxGQtSnaeVXL_Hf8g}{2kG6_YxUQr-4u-DllQsyEQ}{10.124.129.8}{10.124.129.8:9300}{dilmrt}{ml.machine_memory=16656900096, xpack.installed=true, logess=ui, transform.node=true, ml.max_open_jobs=20} Hot threads at 2021-09-03T08:58:37.158Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true: ::: {do-loguip701.prod.mdgapp.net}{OiXW6LsqQwqbTYnX8dr4QA}{kwAVXLvzTQGk6gIsYJhUiQ}{10.124.129.41}{10.124.129.41:9300}{dilmrt}{ml.machine_memory=16656900096, ml.max_open_jobs=20, xpack.installed=true, logess=ui, transform.node=true} Hot threads at 2021-09-03T08:58:37.152Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true: ::: {do-loguip802.prod.mdgapp.net}{Ls-HYzkSS0SFrFwq9lTqXQ}{LsQ1ZqKRSK6_LGMpaZBI1w}{10.125.129.12}{10.125.129.12:9300}{dilmrt}{ml.machine_memory=16656900096, ml.max_open_jobs=20, xpack.installed=true, logess=ui, transform.node=true} Hot threads at 2021-09-03T08:58:37.154Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true: ``` ---
**Author:** ![krish0608](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krish0608/32/6797_2.png) [@krish0608](https://discuss.elastic.co/u/krish0608)\ **Post date:** [September 3, 2021, 1:28pm UTC](https://discuss.elastic.co/t/java-heap-dump-consuming-disk-space/283068/11 "2021-09-03T13:28:00Z")
This is the latest log that we found before dump got created. ```auto [2021-09-03T12:41:05,962][WARN][o.e.c.r.a.AllocationService] [do-loguip701.prod.mdgapp.net] [.async-search][0] marking unavailable shards as stale: [x6Dy3zSpQeC6TfpwKQRHvw] [2021-09-03T12:41:05,977][INFO][o.e.i.s.IndexShard] [do-loguip701.prod.mdgapp.net] [.reporting-2020.09.27][0] primary-replica resync completed with 0 operations [2021-09-03T12:41:07,388][DEBUG][o.e.a.s.TransportSearchAction] [do-loguip701.prod.mdgapp.net] Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[.kibana_task_manager], indicesOptions=IndicesOptions[ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, expand_wildcards_hidden=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=null, scroll=Scroll{keepAlive=5m}, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=null, allowPartialSearchResults=false, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"size":1000,"query":{"bool":{"must":[{"term":{"type":{"value":"task","boost":1.0}}},{"bool":{"must":[{"bool":{"must":[{"bool":{"should":[{"bool":{"must":[{"term":{"task.status":{"value":"idle","boost":1.0}}},{"range":{"task.runAt":{"from":null,"to":"now","include_lower":true,"include_upper":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"bool":{"should":[{"term":{"task.status":{"value":"running","boost":1.0}}},{"term":{"task.status":{"value":"claiming","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"range":{"task.retryAt":{"from":null,"to":"now","include_lower":true,"include_upper":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"should":[{"exists":{"field":"task.schedule","boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"vis_telemetry","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"lens_telemetry","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"actions:.email","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":1,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"actions:.index","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":1,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"actions:.pagerduty","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":1,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"actions:.server-log","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":1,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"actions:.slack","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":1,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"actions:.webhook","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":1,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"actions:.servicenow","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":1,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"actions:.jira","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":1,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"actions:.resilient","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":1,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"actions_telemetry","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"alerting_telemetry","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"alerting:.index-threshold","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"alerting:siem.signals","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"alerting:siem.notifications","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"endpoint:user-artifact-packager","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"alerting:metrics.alert.threshold","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"alerting:metrics.alert.inventory.threshold","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"alerting:logs.alert.document.count","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"alerting:apm.transaction_duration","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"alerting:apm.error_rate","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"apm-telemetry-task","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"alerting:xpack.uptime.alerts.monitorStatus","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"alerting:xpack.uptime.alerts.tls","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"alerting:xpack.uptime.alerts.durationAnomaly","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"filter":[{"bool":{"must_not":[{"bool":{"must":[{"range":{"task.retryAt":{"from":"now","to":null,"include_lower":false,"include_upper":true,"boost":1.0}}}],"should":[{"term":{"task.status":{"value":"running","boost":1.0}}},{"term":{"task.status":{"value":"claiming","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"version":false,"seq_no_primary_term":true,"sort":[{"_score":{"order":"desc"}},{"_script":{"script":{"source":"\nif (doc['task.retryAt'].size()!=0) {\n return doc['task.retryAt'].value.toInstant().toEpochMilli();\n}\nif (doc['task.runAt'].size()!=0) {\n return doc['task.runAt'].value.toInstant().toEpochMilli();\n}\n ","lang":"painless"},"type":"number","order":"asc"}}]}}] [2021-09-03T13:04:55,058][INFO][o.e.c.s.ClusterApplierService] [do-loguip701.prod.mdgapp.net] master node changed {previous [], current [{do-loguip801.prod.mdgapp.net}{BTPvv7gHRtuFwaLYPHQWDA}{Zq2FUExlTByB8Y_dYVPqxA}{10.125.129.41}{10.125.129.41:9300}{dilmrt}{ml.machine_memory=16656900096, ml.max_open_jobs=20, xpack.installed=true, logess=ui, transform.node=true}]}, added {{do-loguip801.prod.mdgapp.net}{BTPvv7gHRtuFwaLYPHQWDA}{Zq2FUExlTByB8Y_dYVPqxA}{10.125.129.41}{10.125.129.41:9300}{dilmrt}{ml.machine_memory=16656900096, ml.max_open_jobs=20, xpack.installed=true, logess=ui, transform.node=true},{do-loguip702.prod.mdgapp.net}{JsxscRxGQtSnaeVXL_Hf8g}{2kG6_YxUQr-4u-DllQsyEQ}{10.124.129.8}{10.124.129.8:9300}{dilmrt}{ml.machine_memory=16656900096, ml.max_open_jobs=20, xpack.installed=true, logess=ui, transform.node=true},{do-loguip802.prod.mdgapp.net}{Ls-HYzkSS0SFrFwq9lTqXQ}{ehURaJcmRQGAYt0sFXfX3g}{10.125.129.12}{10.125.129.12:9300}{dilmrt}{ml.machine_memory=16656900096, ml.max_open_jobs=20, xpack.installed=true, logess=ui, transform.node=true}}, term: 827, version: 326914, reason: ApplyCommitRequest{term=827, version=326914, sourceNode={do-loguip801.prod.mdgapp.net}{BTPvv7gHRtuFwaLYPHQWDA}{Zq2FUExlTByB8Y_dYVPqxA}{10.125.129.41}{10.125.129.41:9300}{dilmrt}{ml.machine_memory=16656900096, ml.max_open_jobs=20, xpack.installed=true, logess=ui, transform.node=true}} [2021-09-03T13:04:55,070][INFO][o.e.c.s.ClusterSettings] [do-loguip701.prod.mdgapp.net] updating [cluster.remote.ctc1.seeds] from [[]] to [["do-logesmasp801.prod.mdgapp.net:9300","do-logesmasp802.prod.mdgapp.net:9300","do-logesmasp803.prod.mdgapp.net:9300"]] [2021-09-03T13:04:55,070][INFO][o.e.c.s.ClusterSettings] [do-loguip701.prod.mdgapp.net] updating [cluster.remote.ptc1.seeds] from [[]] to [["do-logesmasp704.prod.mdgapp.net:9300","do-logesmasp705.prod.mdgapp.net:9300","do-logesmasp706.prod.mdgapp.net:9300"]] [2021-09-03T13:04:55,078][INFO][o.e.c.s.ClusterSettings] [do-loguip701.prod.mdgapp.net] updating [cluster.remote.ctc1.seeds] from [[]] to [["do-logesmasp801.prod.mdgapp.net:9300","do-logesmasp802.prod.mdgapp.net:9300","do-logesmasp803.prod.mdgapp.net:9300"]] [2021-09-03T13:04:55,410][INFO][o.e.c.s.ClusterSettings] [do-loguip701.prod.mdgapp.net] updating [cluster.remote.ptc1.skip_unavailable] from [false] to [true] [2021-09-03T13:04:55,410][INFO][o.e.c.s.ClusterSettings] [do-loguip701.prod.mdgapp.net] updating [cluster.remote.ctc1.skip_unavailable] from [false] to [true] [2021-09-03T13:04:55,783][INFO][o.e.l.LicenseService] [do-loguip701.prod.mdgapp.net] license [602652e7-a795-4bdc-84fc-c0921a27434b] mode [basic] - valid [2021-09-03T13:04:55,785][INFO][o.e.x.s.s.SecurityStatusChangeListener] [do-loguip701.prod.mdgapp.net] Active license is now [BASIC]; Security is disabled [2021-09-03T13:10:50,215][INFO][o.e.m.j.JvmGcMonitorService] [do-loguip701.prod.mdgapp.net] [gc][356] overhead, spent [309ms] collecting in the last [1s] [2021-09-03T13:11:04,049][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [do-loguip701.prod.mdgapp.net] fatal error in thread [elasticsearch[do-loguip701.prod.mdgapp.net][write][T#3]], exiting java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOfRange(Arrays.java:3821) ~[?:?] at java.lang.StringLatin1.newString(StringLatin1.java:764) ~[?:?] ``` ---
**Author:** ![krish0608](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krish0608/32/6797_2.png) [@krish0608](https://discuss.elastic.co/u/krish0608)\ **Post date:** [September 9, 2021, 11:35am UTC](https://discuss.elastic.co/t/java-heap-dump-consuming-disk-space/283068/12 "2021-09-09T11:35:41Z")
can someone provide there input regarding issue.?? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 7, 2021, 11:36am UTC](https://discuss.elastic.co/t/java-heap-dump-consuming-disk-space/283068/13 "2021-10-07T11:36:12Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.