Elastic have some issue about java heap size.
Elastic, Kibana, and Logstash live on 1 server that has 52 GB RAM. We've also tried change the jvm options for elastic from 1 to 40 Gb. But the elastic always shut down all of sudden.
There is no consistent period of the time the elastic alive between variation of jvm option configuration (1-40 gb) that we've tried.
What else do we need to see? thankyou in advance.
[2022-01-21T22:24:59,025][INFO ][o.e.n.Node ] [node-1] starting ...
[2022-01-21T22:24:59,040][INFO ][o.e.x.s.c.f.PersistentCache] [node-1] persistent cache index loaded
[2022-01-21T22:24:59,207][INFO ][o.e.t.TransportService ] [node-1] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2022-01-21T22:24:59,660][INFO ][o.e.c.c.Coordinator ] [node-1] cluster UUID [aZA8EGmTSFKePIDvlJNdhA]
[2022-01-21T22:24:59,754][INFO ][o.e.c.s.MasterService ] [node-1] elected-as-master ([1] nodes joined)[{node-1}{gdIi0ZQcS2yATknSmPpcFA}{JgJXtA6KSiKuR7LCmbgAwQ}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw} elect leader, BECOME_MASTER_TASK, FINISH_ELECTION], term: 118, version: 6944, delta: master node changed {previous , current [{node-1}{gdIi0ZQcS2yATknSmPpcFA}{JgJXtA6KSiKuR7LCmbgAwQ}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}]}
[2022-01-21T22:24:59,848][INFO ][o.e.c.s.ClusterApplierService] [node-1] master node changed {previous , current [{node-1}{gdIi0ZQcS2yATknSmPpcFA}{JgJXtA6KSiKuR7LCmbgAwQ}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}]}, term: 118, version: 6944, reason: Publication{term=118, version=6944}
[2022-01-21T22:24:59,966][INFO ][o.e.h.AbstractHttpServerTransport] [node-1] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
[2022-01-21T22:25:00,106][INFO ][o.e.n.Node ] [node-1] started
[2022-01-21T22:25:00,331][INFO ][o.e.l.LicenseService ] [node-1] license [fd544bc6-f5a6-4797-9903-83b8d624be1a] mode [basic] - valid
[2022-01-21T22:25:00,331][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [node-1] Active license is now [BASIC]; Security is enabled
[2022-01-21T22:25:00,331][INFO ][o.e.g.GatewayService ] [node-1] recovered [31] indices into cluster_state
[2022-01-21T22:25:00,479][ERROR][o.e.x.s.a.e.NativeUsersStore] [node-1] security index is unavailable. short circuiting retrieval of user [logstash_internal]
[2022-01-21T22:25:04,408][INFO ][o.e.c.r.a.AllocationService] [node-1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[datatype-interval][0]]]).
[2022-01-21T22:27:20,649][INFO ][o.e.t.LoggingTaskListener] [node-1] 640 finished with response BulkByScrollResponse[took=2.1s,timed_out=false,sliceId=null,updated=11,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=,search_failures=]
[2022-01-21T22:27:21,175][INFO ][o.e.t.LoggingTaskListener] [node-1] 643 finished with response BulkByScrollResponse[took=3.2s,timed_out=false,sliceId=null,updated=692,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=,search_failures=]
[2022-01-21T22:30:54,591][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][352] overhead, spent [385ms] collecting in the last [1s]
[2022-01-21T22:30:56,600][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][354] overhead, spent [372ms] collecting in the last [1s]
[2022-01-21T22:30:57,911][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][355] overhead, spent [366ms] collecting in the last [1.3s]
[2022-01-21T22:30:59,940][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][357] overhead, spent [369ms] collecting in the last [1s]
[2022-01-21T22:31:01,959][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][359] overhead, spent [371ms] collecting in the last [1s]
[2022-01-21T22:32:46,626][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][461] overhead, spent [325ms] collecting in the last [1s]
[2022-01-21T22:34:33,351][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][567] overhead, spent [319ms] collecting in the last [1s]
[2022-01-21T22:34:35,377][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][569] overhead, spent [348ms] collecting in the last [1s]
[2022-01-21T22:34:36,533][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][570] overhead, spent [341ms] collecting in the last [1.1s]
[2022-01-21T22:34:38,551][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][572] overhead, spent [346ms] collecting in the last [1s]
[2022-01-21T22:34:40,570][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][574] overhead, spent [339ms] collecting in the last [1s]
[2022-01-21T22:34:41,773][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][575] overhead, spent [344ms] collecting in the last [1.2s]
[2022-01-21T22:34:43,807][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][577] overhead, spent [388ms] collecting in the last [1s]
[2022-01-21T22:34:45,838][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][579] overhead, spent [397ms] collecting in the last [1s]
[2022-01-21T22:34:47,088][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][580] overhead, spent [400ms] collecting in the last [1.2s]
[2022-01-21T22:34:49,119][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][582] overhead, spent [401ms] collecting in the last [1s]
[2022-01-21T22:34:51,136][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][584] overhead, spent [405ms] collecting in the last [1s]
[2022-01-21T22:34:52,514][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][585] overhead, spent [420ms] collecting in the last [1.3s]
[2022-01-21T22:34:54,551][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][587] overhead, spent [399ms] collecting in the last [1s]
[2022-01-21T22:34:56,554][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][589] overhead, spent [394ms] collecting in the last [1s]
[2022-01-21T22:34:57,851][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][590] overhead, spent [388ms] collecting in the last [1.3s]
[2022-01-21T22:34:59,882][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][592] overhead, spent [388ms] collecting in the last [1s]
[2022-01-21T22:35:01,905][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][594] overhead, spent [383ms] collecting in the last [1s]
[2022-01-21T22:35:03,203][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][595] overhead, spent [383ms] collecting in the last [1.3s]
[2022-01-21T22:35:05,234][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][597] overhead, spent [384ms] collecting in the last [1s]
[2022-01-21T22:35:05,452][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-1] attempting to trigger G1GC due to high heap usage [40818966528]
[2022-01-21T22:35:05,815][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-1] GC did not bring memory usage down, before [40818966528], after [41203789824], allocations [54], duration [363]
[2022-01-21T22:35:14,034][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][old][600][1] duration [6.6s], collections [1]/[6.6s], total [6.6s]/[6.6s], memory [39.9gb]->[37.5gb]/[40gb], all_pools {[young] [0b]->[0b]/[0b]}{[old] [39.9gb]->[37.5gb]/[40gb]}{[survivor] [0b]->[0b]/[0b]}
[2022-01-21T22:35:14,034][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][600] overhead, spent [6.6s] collecting in the last [6.6s]
[2022-01-21T22:35:14,372][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-1] attempting to trigger G1GC due to high heap usage [40813844168]
[2022-01-21T22:35:14,434][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-1] GC did not bring memory usage down, before [40813844168], after [40870013648], allocations [1], duration [62]
[2022-01-21T22:35:21,683][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][old][602][2] duration [5.7s], collections [1]/[6.6s], total [5.7s]/[12.3s], memory [38.8gb]->[39.6gb]/[40gb], all_pools {[young] [767.9mb]->[0b]/[0b]}{[old] [37.8gb]->[39.6gb]/[40gb]}{[survivor] [256mb]->[0b]/[0b]}
[2022-01-21T22:35:21,683][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-1] attempting to trigger G1GC due to high heap usage [42547741848]
[2022-01-21T22:35:21,695][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][602] overhead, spent [5.9s] collecting in the last [6.6s]
[2022-01-21T22:35:30,119][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-1] GC did not bring memory usage down, before [42547741848], after [42585411840], allocations [1], duration [8436]
[2022-01-21T22:35:30,119][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][POST][/.kibana_task_manager/_update_by_query?ignore_unavailable=true&refresh=true&conflicts=proceed][Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:54221}] took [8435ms] which is above the warn threshold of [5000ms]
[2022-01-21T22:35:30,132][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][GET][/_xpack?accept_enterprise=true][Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:54218}] took [8435ms] which is above the warn threshold of [5000ms]
[2022-01-21T22:35:34,875][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][603] overhead, spent [13.1s] collecting in the last [13.1s]
[2022-01-21T22:35:38,491][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-1] attempting to trigger G1GC due to high heap usage [42598058016]
[2022-01-21T22:35:38,523][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][604] overhead, spent [3.6s] collecting in the last [3.6s]
[2022-01-21T22:35:56,031][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-1] GC did not bring memory usage down, before [42598058016], after [42628950944], allocations [1], duration [17540]
[2022-01-21T22:35:56,051][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][POST][/.kibana_task_manager/_update_by_query?ignore_unavailable=true&refresh=true&conflicts=proceed][Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:54221}] took [17539ms] which is above the warn threshold of [5000ms]
[2022-01-21T22:35:56,044][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][GET][/_xpack][Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:54190}] took [17539ms] which is above the warn threshold of [5000ms]
[2022-01-21T22:35:56,031][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][POST][/.reporting-*/_search][Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:54208}] took [17539ms] which is above the warn threshold of [5000ms]
[2022-01-21T22:52:57,926][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] fatal error in thread [Elasticsearch[node-1][generic][T#26]], exiting
java.lang.OutOfMemoryError: Java heap space
[2022-01-21T22:52:57,926][WARN ][i.n.c.n.NioEventLoop ] [node-1] Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Java heap space
[2022-01-21T22:52:58,099][WARN ][i.n.c.n.NioEventLoop ] [node-1] Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Java heap space
[2022-01-21T22:52:57,926][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] fatal error in thread [ticker-schedule-trigger-engine], exiting
java.lang.OutOfMemoryError: Java heap space
[2022-01-21T22:52:58,078][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] fatal error in thread [Elasticsearch[node-1][refresh][T#1]], exiting
java.lang.OutOfMemoryError: Java heap space
[2022-01-21T22:52:58,181][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] fatal error in thread [Elasticsearch[node-1][transport_worker][T#2]], exiting
java.lang.OutOfMemoryError: Java heap space
[2022-01-21T22:52:58,094][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] fatal error in thread [Elasticsearch[ilm-history-store-flush-scheduler][T#1]], exiting
java.lang.OutOfMemoryError: Java heap space
[2022-01-21T22:52:57,926][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] fatal error in thread [Elasticsearch[node-1][search][T#3]], exiting
java.lang.OutOfMemoryError: Java heap space
[2022-01-21T22:52:58,078][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] fatal error in thread [Connection evictor], exiting
java.lang.OutOfMemoryError: Java heap space