Java Heap dump consuming disk space

Elastic Stack Elasticsearch
1

We have been getting java heap out of memory errors quite regularly, so
this time we decided to take the heap dump and analyze it. But,
unfortunately its tough for us to make anything out of it. So we are
seeking for some help. Also in jvm.options file we have disabled below mentioned parameters:-

#-XX:+UseConcMarkSweepGC
#-XX:CMSInitiatingOccupancyFraction=75
#-XX:+UseCMSInitiatingOccupancyOnly

Just curious to know is these parameter results in creation of Java heap dump files.
We are using 7.9.3 version and also just for info we are using 17 Nodes clusters.

2

What is the output from the _cluster/stats?pretty&human API?

3

@warkolm - Thanks for your revert. Also forget to mention about that we have commented above parameters because recently we have upgraded our cluster from 7.0.1 to 7.9.3 and after upgradation these 3 parameters causing issue and we are not able to restart of Elasticsearch service. Thats why we have commented it out. Here is output of API:-

{
"_nodes" : {
"total" : 4,
"successful" : 4,
"failed" : 0
},
"cluster_name" : "XCS1_LOG_PRODUCTION",
"cluster_uuid" : "DxQkLc6IRFaaKyfsYMnaqw",
"timestamp" : 1630552294934,
"status" : "green",
"indices" : {
"count" : 50,
"shards" : {
"total" : 100,
"primaries" : 50,
"replication" : 1.0,
"index" : {
"shards" : {
"min" : 2,
"max" : 2,
"avg" : 2.0
},
"primaries" : {
"min" : 1,
"max" : 1,
"avg" : 1.0
},
"replication" : {
"min" : 1.0,
"max" : 1.0,
"avg" : 1.0
}
}
},
"docs" : {
"count" : 38690,
"deleted" : 1987
},
"store" : {
"size_in_bytes" : 2207040041,
"reserved_in_bytes" : 0
},
"fielddata" : {
"memory_size_in_bytes" : 0,
"evictions" : 0
},
"query_cache" : {
"memory_size_in_bytes" : 118000,
"total_count" : 1445,
"hit_count" : 129,
"miss_count" : 1316,
"cache_size" : 65,
"cache_count" : 65,
"evictions" : 0
},
"completion" : {
"size_in_bytes" : 0
},
"segments" : {
"count" : 361,
"memory_in_bytes" : 1456984,
"terms_memory_in_bytes" : 1068528,
"stored_fields_memory_in_bytes" : 169816,
"term_vectors_memory_in_bytes" : 0,
"norms_memory_in_bytes" : 68544,
"points_memory_in_bytes" : 0,
"doc_values_memory_in_bytes" : 150096,
"index_writer_memory_in_bytes" : 865280,
"version_map_memory_in_bytes" : 0,
"fixed_bit_set_memory_in_bytes" : 8696,
"max_unsafe_auto_id_timestamp" : 1630546435647,
"file_sizes" : { }
},
"mappings" : {
"field_types" : [
{
"name" : "alias",
"count" : 1,
"index_count" : 1
},
{
"name" : "binary",
"count" : 9,
"index_count" : 1
},
{
"name" : "boolean",
"count" : 106,
"index_count" : 40
},
{
"name" : "byte",
"count" : 33,
"index_count" : 33
},
{
"name" : "date",
"count" : 239,
"index_count" : 49
},
{
"name" : "double",
"count" : 1,
"index_count" : 1
},
{
"name" : "flattened",
"count" : 9,
"index_count" : 1
},
{
"name" : "float",
"count" : 3,
"index_count" : 1
},
{
"name" : "geo_point",
"count" : 1,
"index_count" : 1
},
{
"name" : "integer",
"count" : 29,
"index_count" : 2
},
{
"name" : "ip",
"count" : 2,
"index_count" : 1
},
{
"name" : "keyword",
"count" : 817,
"index_count" : 49
},
{
"name" : "long",
"count" : 150,
"index_count" : 46
},
{
"name" : "nested",
"count" : 16,
"index_count" : 6
},
{
"name" : "object",
"count" : 374,
"index_count" : 50
},
{
"name" : "short",
"count" : 66,
"index_count" : 33
},
{
"name" : "text",
"count" : 319,
"index_count" : 48
}
]
},
"analysis" : {
"char_filter_types" : ,
"tokenizer_types" : ,
"filter_types" : ,
"analyzer_types" : ,
"built_in_char_filters" : ,
"built_in_tokenizers" : ,
"built_in_filters" : ,
"built_in_analyzers" :
}
},
"nodes" : {
"count" : {
"total" : 4,
"coordinating_only" : 0,
"data" : 4,
"ingest" : 4,
"master" : 4,
"ml" : 4,
"remote_cluster_client" : 4,
"transform" : 4,
"voting_only" : 0
},
"versions" : [
"7.9.3"
],
"os" : {
"available_processors" : 16,
"allocated_processors" : 16,
"names" : [
{
"name" : "Linux",
"count" : 4
}
],
"pretty_names" : [
{
"pretty_name" : "Red Hat Enterprise Linux Server 7.9 (Maipo)",
"count" : 4
}
],
"mem" : {
"total_in_bytes" : 66627600384,
"free_in_bytes" : 12966031360,
"used_in_bytes" : 53661569024,
"free_percent" : 19,
"used_percent" : 81
}
},
"process" : {
"cpu" : {
"percent" : 0
},
"open_file_descriptors" : {
"min" : 486,
"max" : 560,
"avg" : 511
}
},
"jvm" : {
"max_uptime_in_millis" : 84338505,
"versions" : [
{
"version" : "15",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "15+36-1562",
"vm_vendor" : "Oracle Corporation",
"bundled_jdk" : true,
"using_bundled_jdk" : true,
"count" : 4
}
],
"mem" : {
"heap_used_in_bytes" : 15199421360,
"heap_max_in_bytes" : 33319550976
},
"threads" : 274
},
"fs" : {
"total_in_bytes" : 42907729920,
"free_in_bytes" : 25755017216,
"available_in_bytes" : 25755017216
},
"plugins" : ,
"network_types" : {
"transport_types" : {
"security4" : 4
},
"http_types" : {
"security4" : 4
}
},
"discovery_types" : {
"zen" : 4
},
"packaging_types" : [
{
"flavor" : "default",
"type" : "rpm",
"count" : 4
}
],
"ingest" : {
"number_of_pipelines" : 4,
"processor_stats" : {
"date" : {
"count" : 0,
"failed" : 0,
"current" : 0,
"time_in_millis" : 0
},
"gsub" : {
"count" : 0,
"failed" : 0,
"current" : 0,
"time_in_millis" : 0
},
"rename" : {
"count" : 0,
"failed" : 0,
"current" : 0,
"time_in_millis" : 0
},
"script" : {
"count" : 0,
"failed" : 0,
"current" : 0,
"time_in_millis" : 0
},
"set" : {
"count" : 0,
"failed" : 0,
"current" : 0,
"time_in_millis" : 0
}
}
}
}
}

4

Please format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you :slight_smile:

Is your heap size set to ~15GB per node?

5

Yes i guess. Is something wrong i have configured.?

6

Just checking.

What do your Elasticsearch logs show in terms of the OOM?

7 
[2021-08-30T22:56:36,561][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [do-loguip701.prod.mdgapp.net] GC did bring memory usage down, before [7834664800], after [4785620880], allocations [1], duration [82]
[2021-08-30T23:21:43,040][INFO ][o.e.c.s.ClusterApplierService] [do-loguip701.prod.mdgapp.net] removed {{do-loguip801.prod.mdgapp.net}{BTPvv7gHRtuFwaLYPHQWDA}{3ZBGGg94Rb2psre_Q0iegA}{10.125.129.41}{10.125.129.41:9300}{dilmrt}{ml.machine_memory=16656900096, ml.max_open_jobs=20, xpack.installed=true, logess=ui, transform.node=true}}, term: 811, version: 319691, reason: ApplyCommitRequest{term=811, version=319691, sourceNode={do-loguip702.prod.mdgapp.net}{JsxscRxGQtSnaeVXL_Hf8g}{qF_Kh0bIQj2drsxZ4JpP3Q}{10.124.129.8}{10.124.129.8:9300}{dilmrt}{ml.machine_memory=16656900096, ml.max_open_jobs=20, xpack.installed=true, logess=ui, transform.node=true}}
Caused by: java.lang.IllegalArgumentException: Text fields are not optimised for operations that require per-document field data like aggre...

[2021-08-30T17:36:49,398][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [do-loguip701.prod.mdgapp.net] attempting to trigger G1GC due to high heap usage [7913509344]
[2021-08-30T17:41:50,401][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [do-loguip701.prod.mdgapp.net] attempting to trigger G1GC due to high heap usage [7941661376]

 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks
        at org.elasticsearch.tasks.TaskManager$CancellableTaskHolder.registerChildNode(TaskManager.java:522) ~[elasticsearch-7.9.3.jar:7.9.3]
        at org.elasticsearch.tasks.TaskManager.registerChildNode(TaskManager.java:213) ~[elasticsearch-7.9.3.jar:7.9.3]
        at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:626) ~[elasticsearch-7.9.3.jar:7.9.3]
        ... 66 more
[2021-08-31T22:47:52,610][INFO ][o.e.t.TaskCancellationService] [do-loguip701.prod.mdgapp.net] failed to remove the parent ban for task OiXW6LsqQwqbTYnX8dr4QA:296342 on node {do-logwarmp801.prod.mdgapp.net}{v-eOjR5rQRqzD1bVuPSgrA}{C-BLLoRpQJS3TdX-g1yKUw}{10.125.129.53}{10.125.129.53:9300}{dlrt}{ml.machine_memory=18769842176, ml.max_open_jobs=20, xpack.installed=true, logess=warm, transform.node=true}
[2021-08-31T22:47:52,872][WARN ][o.e.t.TaskCancellationService] [do-loguip701.prod.mdgapp.net] Cannot send ban for tasks with the parent [OiXW6LsqQwqbTYnX8dr4QA:296682] to the node [{do-loghotp703.prod.mdgapp.net}{WxRnUg8XS_2ml1Ndwd9ENw}{hqHF-GKWRImltSr3JaqiLA}{10.124.129.58}{10.124.129.58:9300}{dilrt}{ml.machine_memory=33566654464, ml.max_open_jobs=20, xpack.installed=true, logess=hot, transform.node=true}]
[2021-08-31T22:47:52,883][INFO ][o.e.t.TaskCancellationService] [do-loguip701.prod.mdgapp.net] failed to remove the parent ban for task OiXW6LsqQwqbTYnX8dr4QA:296682 on node {do-loghotp703.prod.mdgapp.net}{WxRnUg8XS_2ml1Ndwd9ENw}{hqHF-GKWRImltSr3JaqiLA}{10.124.129.58}{10.124.129.58:9300}{dilrt}{ml.machine_memory=33566654464, ml.max_open_jobs=20, xpack.installed=true, logess=hot, transform.node=true}
[2021-08-31T22:52:50,765][WARN ][o.e.t.TaskCancellationService] [do-loguip701.prod.mdgapp.net] Cannot send ban for tasks with the parent [OiXW6LsqQwqbTYnX8dr4QA:312162] to the node [{do-loghotp701.prod.mdgapp.net}{j91jdhtGQUG9SWG4L05LHA}{ugHoG47aS_-UA4C3PtzviQ}{10.124.129.26}{10.124.129.26:9300}{dilrt}{ml.machine_memory=33566654464, ml.max_open_jobs=20, xpack.installed=true, logess=hot, transform.node=true}]
[2021-08-31T22:52:50,781][INFO ][o.e.t.TaskCancellationService] [do-loguip701.prod.mdgapp.net] failed to remove the parent ban for task OiXW6LsqQwqbTYnX8dr4QA:312162 on node {do-loghotp701.prod.mdgapp.net}{j91jdhtGQUG9SWG4L05LHA}{ugHoG47aS_-UA4C3PtzviQ}{10.124.129.26}{10.124.129.26:9300}{dilrt}{ml.machine_memory=33566654464, ml.max_open_jobs=20, xpack.installed=true, logess=hot, transform.node=true}
[2021-08-31T22:52:51,373][WARN ][o.e.t.TaskCancellationService] [do-loguip701.prod.mdgapp.net] Cannot send ban for tasks with the parent [OiXW6LsqQwqbTYnX8dr4QA:312268] to the node [{do-loghotp701.prod.mdgapp.net}{j91jdhtGQUG9SWG4L05LHA}{ugHoG47aS_-UA4C3PtzviQ}{10.124.129.26}{10.124.129.26:9300}{dilrt}{ml.machine_memory=33566654464, ml.max_open_jobs=20, xpack.installed=true, logess=hot, transform.node=true}]
[2021-08-31T22:52:51,373][WARN ][o.e.t.TaskCancellationService] [do-loguip701.prod.mdgapp.net] Cannot send ban for tasks with the parent [OiXW6LsqQwqbTYnX8dr4QA:312268] to the node [{do-logwarmp804.prod.mdgapp.net}{Iur-7puaTlyCg9omtKQ1SA}{Qj7eA5UORRGyFVrYsIhkuw}{10.125.129.22}{10.125.129.22:9300}{dlrt}{ml.machine_memory=18769842176, ml.max_open_jobs=20, xpack.installed=true, logess=warm, transform.node=true}]
[2021-08-31T22:52:51,373][DEBUG][o.e.a.s.TransportSearchAction] [do-loguip701.prod.mdgapp.net] [logstash-requestjail-2021.08.31-000498][0], node[dtOI6jD2RCSubXL4wS1qrw], [P], s[STARTED], a[id=R3pdffNkTn6DWgVKje0zhg]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[*:logstash-requestjail*], indicesOptions=IndicesOptions[ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, expand_wildcards_hidden=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=false], types=[], routing='null', preference='1630450359304', requestCache=true, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=64, preFilterShardSize=1, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=false, source={"size":500,"query":{"bool":{"must":[{"query_string":{"query":"*","default_field":"*","fields":[],"type":"best_fields","default_operator":"or","max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"analyze_wildcard":true,"time_zone":"America/Denver","escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}},{"match_all":{"boost":1.0}}],"filter":[{"match_phrase":{"marketer":{"query":"TC","slop":0,"zero_terms_query":"NONE","boost":1.0}}},{"match_phrase":{"environment.keyword":{"query":"production","slop":0,"zero_terms_query":"NONE","boost":1.0}}},{"range":{"@timestamp":{"from":"2021-08-31T06:00:00.000Z","to":"2021-09-01T05:59:59.999Z","include_lower":true,"include_upper":true,"format":"strict_date_optional_time","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"version":true,"_source":{"includes":[],"excludes":[]},"stored_fields":"*","docvalue_fields":[{"field":"@timestamp","format":"date_time"}],"script_fields":{},"sort":[],"track_total_hits":2147483647}}] lastShard [true]
org.elasticsearch.transport.TransportException: failure to send
        at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:659) [elasticsearch-7.9.3.jar:7.9.3]
        at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:703) [elasticsearch-7.9.3.jar:7.9.3]
        at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:695) [elasticsearch-7.9.3.jar:7.9.3]
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3]
        at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3]

is this helpful for you?

8 
Caused by: org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks
        at org.elasticsearch.tasks.TaskManager$CancellableTaskHolder.registerChildNode(TaskManager.java:522) ~[elasticsearch-7.9.3.jar:7.9.3]
        at org.elasticsearch.tasks.TaskManager.registerChildNode(TaskManager.java:213) ~[elasticsearch-7.9.3.jar:7.9.3]
        at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:626) ~[elasticsearch-7.9.3.jar:7.9.3]
        ... 66 more
[2021-09-02T15:11:33,797][INFO ][o.e.m.j.JvmGcMonitorService] [do-loguip702.prod.mdgapp.net] [gc][127471] overhead, spent [278ms] collecting in the last [1s]
[2021-09-02T15:24:17,501][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [do-loguip702.prod.mdgapp.net] attempting to trigger G1GC due to high heap usage [8109806080]
[2021-09-02T15:24:17,526][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [do-loguip702.prod.mdgapp.net] GC did bring memory usage down, before [8109806080], after [5376480440], allocations [37], duration [26]
[2021-09-02T15:24:19,452][WARN ][o.e.m.j.JvmGcMonitorService] [do-loguip702.prod.mdgapp.net] [gc][128236] overhead, spent [515ms] collecting in the last [1s]
[2021-09-02T15:24:20,453][INFO ][o.e.m.j.JvmGcMonitorService] [do-loguip702.prod.mdgapp.net] [gc][128237] overhead, spent [439ms] collecting in the last [1s]
[2021-09-02T15:24:24,859][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [do-loguip702.prod.mdgapp.net] fatal error in thread [elasticsearch[do-loguip702.prod.mdgapp.net][write][T#3]], exiting
java.lang.OutOfMemoryError: Java heap space

Here is latest got i have just recieved.

9

Thanks. What's in hot threads on that node?

10

@warkolm -I think you are lookint for _nodes/hot_threads output. Please correct if am wrong.

::: {do-loguip801.prod.mdgapp.net}{BTPvv7gHRtuFwaLYPHQWDA}{Zq2FUExlTByB8Y_dYVPqxA}{10.125.129.41}{10.125.129.41:9300}{dilmrt}{ml.machine_memory=16656900096, ml.max_open_jobs=20, xpack.installed=true, logess=ui, transform.node=true}
   Hot threads at 2021-09-03T08:58:37.150Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

::: {do-loguip702.prod.mdgapp.net}{JsxscRxGQtSnaeVXL_Hf8g}{2kG6_YxUQr-4u-DllQsyEQ}{10.124.129.8}{10.124.129.8:9300}{dilmrt}{ml.machine_memory=16656900096, xpack.installed=true, logess=ui, transform.node=true, ml.max_open_jobs=20}
   Hot threads at 2021-09-03T08:58:37.158Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

::: {do-loguip701.prod.mdgapp.net}{OiXW6LsqQwqbTYnX8dr4QA}{kwAVXLvzTQGk6gIsYJhUiQ}{10.124.129.41}{10.124.129.41:9300}{dilmrt}{ml.machine_memory=16656900096, ml.max_open_jobs=20, xpack.installed=true, logess=ui, transform.node=true}
   Hot threads at 2021-09-03T08:58:37.152Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

::: {do-loguip802.prod.mdgapp.net}{Ls-HYzkSS0SFrFwq9lTqXQ}{LsQ1ZqKRSK6_LGMpaZBI1w}{10.125.129.12}{10.125.129.12:9300}{dilmrt}{ml.machine_memory=16656900096, ml.max_open_jobs=20, xpack.installed=true, logess=ui, transform.node=true}
   Hot threads at 2021-09-03T08:58:37.154Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
11

This is the latest log that we found before dump got created.

[2021-09-03T12:41:05,962][WARN ][o.e.c.r.a.AllocationService] [do-loguip701.prod.mdgapp.net] [.async-search][0] marking unavailable shards as stale: [x6Dy3zSpQeC6TfpwKQRHvw]
[2021-09-03T12:41:05,977][INFO ][o.e.i.s.IndexShard       ] [do-loguip701.prod.mdgapp.net] [.reporting-2020.09.27][0] primary-replica resync completed with 0 operations
[2021-09-03T12:41:07,388][DEBUG][o.e.a.s.TransportSearchAction] [do-loguip701.prod.mdgapp.net] Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[.kibana_task_manager], indicesOptions=IndicesOptions[ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, expand_wildcards_hidden=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=null, scroll=Scroll{keepAlive=5m}, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=null, allowPartialSearchResults=false, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"size":1000,"query":{"bool":{"must":[{"term":{"type":{"value":"task","boost":1.0}}},{"bool":{"must":[{"bool":{"must":[{"bool":{"should":[{"bool":{"must":[{"term":{"task.status":{"value":"idle","boost":1.0}}},{"range":{"task.runAt":{"from":null,"to":"now","include_lower":true,"include_upper":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"bool":{"should":[{"term":{"task.status":{"value":"running","boost":1.0}}},{"term":{"task.status":{"value":"claiming","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"range":{"task.retryAt":{"from":null,"to":"now","include_lower":true,"include_upper":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"should":[{"exists":{"field":"task.schedule","boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"vis_telemetry","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"lens_telemetry","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"actions:.email","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":1,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"actions:.index","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":1,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"actions:.pagerduty","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":1,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"actions:.server-log","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":1,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"actions:.slack","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":1,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"actions:.webhook","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":1,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"actions:.servicenow","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":1,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"actions:.jira","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":1,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"actions:.resilient","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":1,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"actions_telemetry","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"alerting_telemetry","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"alerting:.index-threshold","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"alerting:siem.signals","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"alerting:siem.notifications","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"endpoint:user-artifact-packager","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"alerting:metrics.alert.threshold","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"alerting:metrics.alert.inventory.threshold","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"alerting:logs.alert.document.count","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"alerting:apm.transaction_duration","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"alerting:apm.error_rate","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"apm-telemetry-task","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"alerting:xpack.uptime.alerts.monitorStatus","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"alerting:xpack.uptime.alerts.tls","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"must":[{"term":{"task.taskType":{"value":"alerting:xpack.uptime.alerts.durationAnomaly","boost":1.0}}},{"range":{"task.attempts":{"from":null,"to":3,"include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"filter":[{"bool":{"must_not":[{"bool":{"must":[{"range":{"task.retryAt":{"from":"now","to":null,"include_lower":false,"include_upper":true,"boost":1.0}}}],"should":[{"term":{"task.status":{"value":"running","boost":1.0}}},{"term":{"task.status":{"value":"claiming","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"version":false,"seq_no_primary_term":true,"sort":[{"_score":{"order":"desc"}},{"_script":{"script":{"source":"\nif (doc['task.retryAt'].size()!=0) {\n  return doc['task.retryAt'].value.toInstant().toEpochMilli();\n}\nif (doc['task.runAt'].size()!=0) {\n  return doc['task.runAt'].value.toInstant().toEpochMilli();\n}\n    ","lang":"painless"},"type":"number","order":"asc"}}]}}] 

[2021-09-03T13:04:55,058][INFO ][o.e.c.s.ClusterApplierService] [do-loguip701.prod.mdgapp.net] master node changed {previous [], current [{do-loguip801.prod.mdgapp.net}{BTPvv7gHRtuFwaLYPHQWDA}{Zq2FUExlTByB8Y_dYVPqxA}{10.125.129.41}{10.125.129.41:9300}{dilmrt}{ml.machine_memory=16656900096, ml.max_open_jobs=20, xpack.installed=true, logess=ui, transform.node=true}]}, added {{do-loguip801.prod.mdgapp.net}{BTPvv7gHRtuFwaLYPHQWDA}{Zq2FUExlTByB8Y_dYVPqxA}{10.125.129.41}{10.125.129.41:9300}{dilmrt}{ml.machine_memory=16656900096, ml.max_open_jobs=20, xpack.installed=true, logess=ui, transform.node=true},{do-loguip702.prod.mdgapp.net}{JsxscRxGQtSnaeVXL_Hf8g}{2kG6_YxUQr-4u-DllQsyEQ}{10.124.129.8}{10.124.129.8:9300}{dilmrt}{ml.machine_memory=16656900096, ml.max_open_jobs=20, xpack.installed=true, logess=ui, transform.node=true},{do-loguip802.prod.mdgapp.net}{Ls-HYzkSS0SFrFwq9lTqXQ}{ehURaJcmRQGAYt0sFXfX3g}{10.125.129.12}{10.125.129.12:9300}{dilmrt}{ml.machine_memory=16656900096, ml.max_open_jobs=20, xpack.installed=true, logess=ui, transform.node=true}}, term: 827, version: 326914, reason: ApplyCommitRequest{term=827, version=326914, sourceNode={do-loguip801.prod.mdgapp.net}{BTPvv7gHRtuFwaLYPHQWDA}{Zq2FUExlTByB8Y_dYVPqxA}{10.125.129.41}{10.125.129.41:9300}{dilmrt}{ml.machine_memory=16656900096, ml.max_open_jobs=20, xpack.installed=true, logess=ui, transform.node=true}}
[2021-09-03T13:04:55,070][INFO ][o.e.c.s.ClusterSettings  ] [do-loguip701.prod.mdgapp.net] updating [cluster.remote.ctc1.seeds] from [[]] to [["do-logesmasp801.prod.mdgapp.net:9300","do-logesmasp802.prod.mdgapp.net:9300","do-logesmasp803.prod.mdgapp.net:9300"]]
[2021-09-03T13:04:55,070][INFO ][o.e.c.s.ClusterSettings  ] [do-loguip701.prod.mdgapp.net] updating [cluster.remote.ptc1.seeds] from [[]] to [["do-logesmasp704.prod.mdgapp.net:9300","do-logesmasp705.prod.mdgapp.net:9300","do-logesmasp706.prod.mdgapp.net:9300"]]
[2021-09-03T13:04:55,078][INFO ][o.e.c.s.ClusterSettings  ] [do-loguip701.prod.mdgapp.net] updating [cluster.remote.ctc1.seeds] from [[]] to [["do-logesmasp801.prod.mdgapp.net:9300","do-logesmasp802.prod.mdgapp.net:9300","do-logesmasp803.prod.mdgapp.net:9300"]]
[2021-09-03T13:04:55,410][INFO ][o.e.c.s.ClusterSettings  ] [do-loguip701.prod.mdgapp.net] updating [cluster.remote.ptc1.skip_unavailable] from [false] to [true]
[2021-09-03T13:04:55,410][INFO ][o.e.c.s.ClusterSettings  ] [do-loguip701.prod.mdgapp.net] updating [cluster.remote.ctc1.skip_unavailable] from [false] to [true]
[2021-09-03T13:04:55,783][INFO ][o.e.l.LicenseService     ] [do-loguip701.prod.mdgapp.net] license [602652e7-a795-4bdc-84fc-c0921a27434b] mode [basic] - valid
[2021-09-03T13:04:55,785][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [do-loguip701.prod.mdgapp.net] Active license is now [BASIC]; Security is disabled
[2021-09-03T13:10:50,215][INFO ][o.e.m.j.JvmGcMonitorService] [do-loguip701.prod.mdgapp.net] [gc][356] overhead, spent [309ms] collecting in the last [1s]
[2021-09-03T13:11:04,049][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [do-loguip701.prod.mdgapp.net] fatal error in thread [elasticsearch[do-loguip701.prod.mdgapp.net][write][T#3]], exiting
java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOfRange(Arrays.java:3821) ~[?:?]
        at java.lang.StringLatin1.newString(StringLatin1.java:764) ~[?:?]
12

can someone provide there input regarding issue.??

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.