# Elastic node crashes after kibana query - with java.lang.OutOfMemoryError: Java heap space **URL:** \ **Category:** Elasticsearch\ **Created:** [November 8, 2019, 3:17pm UTC](https://discuss.elastic.co/t/elastic-node-crashes-after-kibana-query-with-java-lang-outofmemoryerror-java-heap-space/207134 "2019-11-08T15:17:24Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Petr.Simik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/petr.simik/32/38082_2.png) [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\ **Post date:** [November 8, 2019, 3:17pm UTC](https://discuss.elastic.co/t/elastic-node-crashes-after-kibana-query-with-java-lang-outofmemoryerror-java-heap-space/207134/1 "2019-11-08T15:17:24Z")
the Query in Kibana is this simple: Metrics: unique count(device\_id.keyword) Buckets: subscription\_code.keyword My cluster is 8 nodes 8core/64G RAM, 31G Heap, Elastic version 6.4.2 ![heap_problem3](https://us1.discourse-cdn.com/elastic/original/3X/e/e/eeab1d0cd1a7fca34b8b4eff17482345de8fded0.png) ![heap1](https://us1.discourse-cdn.com/elastic/original/3X/6/8/68373d8e63729752498b8b66c7f0b7b0f4479358.png) ![heap2](https://us1.discourse-cdn.com/elastic/original/3X/c/0/c03e4097c1e1272020346cc0df39d281b8be46f9.png) > [2019-11-08T15:19:22,118][WARN][o.e.m.j.JvmGcMonitorService] [elastic\_node5] [gc][old][10741][6] duration [29.5s], collections [2]/[29.6s], total [29.5s]/[29.9s], memory [30.3gb]-\>[30.8gb]/[30.9gb], all\_pools {[young] [398.1mb]-\>[532.5mb]/[ > 532.5mb]}{[survivor] [66.5mb]-\>[25.4mb]/[66.5mb]}{[old] [29.8gb]-\>[30.3gb]/[30.3gb]} > [2019-11-08T15:19:22,123][WARN][o.e.m.j.JvmGcMonitorService] [elastic\_node5] [gc][10741] overhead, spent [29.5s] collecting in the last [29.6s] > [2019-11-08T15:20:21,437][WARN][o.e.m.j.JvmGcMonitorService] [elastic\_node5] [gc][old][10742][11] duration [1m], collections [5]/[1m], total [1m]/[1.5m], memory [30.8gb]-\>[30.9gb]/[30.9gb], all\_pools {[young] [532.5mb]-\>[532.5mb]/[532.5mb]} > {[survivor] [25.4mb]-\>[65.5mb]/[66.5mb]}{[old] [30.3gb]-\>[30.3gb]/[30.3gb]} > [2019-11-08T15:20:21,437][WARN][o.e.m.j.JvmGcMonitorService] [elastic\_node5] [gc][10742] overhead, spent [1m] collecting in the last [1m] > [2019-11-08T15:28:20,244][ERROR][o.e.x.m.c.n.NodeStatsCollector] [elastic\_node5] collector [node\_stats] timed out when collecting data > [2019-11-08T15:28:21,143][WARN][o.e.x.s.t.n.SecurityNetty4ServerTransport] [elastic\_node5] send message failed [channel: NettyTcpChannel{localAddress=/1.1.1.1:9300, remoteAddress=/12.2.2.2:58926}] > java.nio.channels.ClosedChannelException: null > at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?] > [2019-11-08T15:28:21,144][WARN][o.e.x.s.t.n.SecurityNetty4ServerTransport] [elastic\_node5] send message failed [channel: NettyTcpChannel{localAddress=/1.1.1.1:9300, remoteAddress=/12.2.2.2:58926}] > java.nio.channels.ClosedChannelException: null > at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?] > [2019-11-08T15:28:21,144][WARN][o.e.x.s.t.n.SecurityNetty4ServerTransport] [elastic\_node5] send message failed [channel: NettyTcpChannel{localAddress=/1.1.1.1:9300, remoteAddress=/12.2.2.2:58926}] > java.nio.channels.ClosedChannelException: null > at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?] > [2019-11-08T15:28:21,144][WARN][o.e.x.s.t.n.SecurityNetty4ServerTransport] [elastic\_node5] send message failed [channel: NettyTcpChannel{localAddress=/1.1.1.1:9300, remoteAddress=/12.2.2.2:58926}] > java.nio.channels.ClosedChannelException: null > at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?] > [2019-11-08T15:28:20,404][WARN][o.e.x.s.t.n.SecurityNetty4ServerTransport] [elastic\_node5] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=/3.3.3.3:38318}] > java.nio.channels.ClosedChannelException: null > at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?] > [2019-11-08T15:28:21,066][ERROR][o.e.i.e.Engine] [elastic\_node5] [index\_norm\_ekt\_00847][3] already closed by tragic event on the index writer > > 2019-11-08T15:28:21,066][ERROR][o.e.i.e.Engine] [elastic\_node5] [index\_norm\_ekt\_00847][3] already closed by tragic event on the index writer > java.lang.OutOfMemoryError: Java heap space ---
**Author:** ![Petr.Simik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/petr.simik/32/38082_2.png) [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\ **Post date:** [November 8, 2019, 6:42pm UTC](https://discuss.elastic.co/t/elastic-node-crashes-after-kibana-query-with-java-lang-outofmemoryerror-java-heap-space/207134/2 "2019-11-08T18:42:58Z")
Do you have any idea experience why this happend? User query crashes the server, why elastic does not protect against such queries? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 9, 2019, 5:13am UTC](https://discuss.elastic.co/t/elastic-node-crashes-after-kibana-query-with-java-lang-outofmemoryerror-java-heap-space/207134/3 "2019-11-09T05:13:26Z")
What does your node configuration look like? Do you have any non-default settings? ---
**Author:** ![Petr.Simik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/petr.simik/32/38082_2.png) [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\ **Post date:** [November 9, 2019, 6:04am UTC](https://discuss.elastic.co/t/elastic-node-crashes-after-kibana-query-with-java-lang-outofmemoryerror-java-heap-space/207134/4 "2019-11-09T06:04:35Z")
@Christian_Dahlqvist good question this is my config (same on all nodes) > node01:/etc/elasticsearch# cat elasticsearch.yml |grep -v ^# > cluster.name: corp-cz-cem > node.name: node01-prahkz > path.data: /data/elasticsearch > path.logs: /data/elasticsearch/log > bootstrap.memory\_lock: true > network.host: 0.0.0.0 > discovery.zen.ping.unicast.hosts: ["node13-prahkzcorp", "node14-prahkzcorp", "node15-prahkzcorp", "node16-prahkzcorp"] > node01-prahkz:/etc/elasticsearch# cat jvm.options |grep -v ^# > > -Xms31g > -Xmx31g > > -XX:+UseConcMarkSweepGC > -XX:CMSInitiatingOccupancyFraction=75 > -XX:+UseCMSInitiatingOccupancyOnly > > -XX:+AlwaysPreTouch > > -Xss1m > > -Djava.awt.headless=true > > -Dfile.encoding=UTF-8 > > -Djna.nosys=true > > -XX:-OmitStackTraceInFastThrow > > -Dio.netty.noUnsafe=true > -Dio.netty.noKeySetOptimization=true > -Dio.netty.recycler.maxCapacityPerThread=0 > > -Dlog4j.shutdownHookEnabled=false > -Dlog4j2.disable.jmx=true > > -Djava.io.tmpdir=${ES\_TMPDIR} > > -XX:+HeapDumpOnOutOfMemoryError > > -XX:HeapDumpPath=/data/elasticsearch/log > > -XX:ErrorFile=/var/log/elasticsearch/hs\_err\_pid%p.log > > 8:-XX:+PrintGCDetails > 8:-XX:+PrintGCDateStamps > 8:-XX:+PrintTenuringDistribution > 8:-XX:+PrintGCApplicationStoppedTime > 8:-Xloggc:/var/log/elasticsearch/gc.log > 8:-XX:+UseGCLogFileRotation > 8:-XX:NumberOfGCLogFiles=32 > 8:-XX:GCLogFileSize=64m > > 9-:-Xlog:gc\*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m > 9-:-Djava.locale.providers=COMPAT > > 10-:-XX:UseAVX=2 ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 9, 2019, 6:17am UTC](https://discuss.elastic.co/t/elastic-node-crashes-after-kibana-query-with-java-lang-outofmemoryerror-java-heap-space/207134/5 "2019-11-09T06:17:44Z")
What is the settings for those metrics and buckets in Kibana? What is the cardinality of the device\_id and subscription\_code fields? Are you using default dynamic mappings? ---
**Author:** ![Petr.Simik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/petr.simik/32/38082_2.png) [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\ **Post date:** [November 11, 2019, 7:32am UTC](https://discuss.elastic.co/t/elastic-node-crashes-after-kibana-query-with-java-lang-outofmemoryerror-java-heap-space/207134/6 "2019-11-11T07:32:48Z")
> [@Christian\_Dahlqvist](#): > > d and subscription\_code fields Unique count of device\_id: 280k Count: 28 mil Unique count of subscription\_code.keyword: 240k this are number among whole day, I need to breakdown it to time windows. 3-4h long time buckets. However in peak there could be the same amount of numbers provided. what other approach would you suggest to calculate this use-case? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 9, 2019, 7:32am UTC](https://discuss.elastic.co/t/elastic-node-crashes-after-kibana-query-with-java-lang-outofmemoryerror-java-heap-space/207134/7 "2019-12-09T07:32:49Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.