# Aggregation query making elastic datanode offline due to long GC **URL:** \ **Category:** Elasticsearch\ **Created:** [November 10, 2017, 12:06pm UTC](https://discuss.elastic.co/t/aggregation-query-making-elastic-datanode-offline-due-to-long-gc/107179 "2017-11-10T12:06:30Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![chitender](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chitender/32/22621_2.png) [@chitender](https://discuss.elastic.co/u/chitender)\ **Post date:** [November 10, 2017, 12:06pm UTC](https://discuss.elastic.co/t/aggregation-query-making-elastic-datanode-offline-due-to-long-gc/107179/1 "2017-11-10T12:06:30Z")
we are running 5 node cluster version 2.3.5. one as a client node and 4 node configured to work as Data and Master nodes. all nodes has 32GB heap size. below are the JAVA OPTS with which process is configured. `-Xms32g -Xmx32g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true` now we are running below search query on index XYZ of 97GB which is making ES nodes to go offline because of long GC run. `{"from": 0,"size": 0,"query": {"bool": {"must": {"bool": {"must": [{"match": {"id": {"query": "9999","type": "phrase"}}},{"match": {"cltp": {"query": "encounters","type": "phrase"}}}]}}}},"_source": {"includes": ["COUNT"],"excludes": []},"aggregations": {"eid": {"terms": {"field": "eid","size": 200},"aggregations": {"COUNT(DISTINCT eid)": {"cardinality": {"field": "eid","precision_threshold": 40000}}}}}}` below is the shard distribution of index. DN1: Replica 0,2,3 DN2: Primary 0,1,4 DN3: Primary 2,3 DN4:Replica 1,4 during the query node DN1 and DN4 falling out of cluster due to long GC cycles. below are logs for same. DN1: [2017-11-10 08:27:25,898][INFO][monitor.jvm] [10.0.1.134] [gc][old][1076][3] duration [7.4s], collections [1]/[7.4s], total [7.4s]/[16.9s], memory [31.5gb]-\>[31.8gb]/[31.8gb], all\_pools {[young] [614.5mb]-\>[865.3mb]/[865.3mb]}{[survivor] [0b]-\>[107.4mb]/[108.1mb]}{[old] [30.9gb]-\>[30.9gb]/[30.9gb]} [2017-11-10 08:28:17,615][INFO][monitor.jvm] [10.0.1.134] [gc][old][1077][18] duration [1.2m], collections [15]/[1.2m], total [1.2m]/[1.5m], memory [31.8gb]-\>[31.8gb]/[31.8gb], all\_pools {[young] [865.3mb]-\>[865.3mb]/[865.3mb]}{[survivor] [107.4mb]-\>[108mb]/[108.1mb]}{[old] [30.9gb]-\>[30.9gb]/[30.9gb]} [2017-11-10 08:28:49,877][INFO][monitor.jvm] [10.0.1.134] [gc][old][1080][26] duration [12.1s], collections [2]/[12.1s], total [12.1s]/[2.1m], memory [31.8gb]-\>[31.8gb]/[31.8gb], all\_pools {[young] [865.3mb]-\>[865.3mb]/[865.3mb]}{[survivor] [108mb]-\>[108mb]/[108.1mb]}{[old] [30.9gb]-\>[30.9gb]/[30.9gb]} DN4: [2017-11-10 08:26:36,566][WARN][monitor.jvm] [10.0.1.95] [gc][young][1074][32] duration [2.3s], collections [1]/[2.4s], total [2.3s]/[11.2s], memory [19.6gb]-\>[20.4gb]/[31.8gb], all\_pools {[young] [17.5mb]-\>[8.7mb]/[865.3mb]}{[survivor] [108.1mb]-\>[108.1mb]/[108.1mb]}{[old] [19.5gb]-\>[20.3gb]/[30.9gb]} [2017-11-10 08:26:38,016][WARN][monitor.jvm] [10.0.1.95] [gc][young][1075][33] duration [1.3s], collections [1]/[1.4s], total [1.3s]/[12.5s], memory [20.4gb]-\>[21.3gb]/[31.8gb], all\_pools {[young] [8.7mb]-\>[26mb]/[865.3mb]}{[survivor] [108.1mb]-\>[108.1mb]/[108.1mb]}{[old] [20.3gb]-\>[21.2gb]/[30.9gb]} [2017-11-10 08:26:39,693][WARN][monitor.jvm] [10.0.1.95] [gc][young][1076][34] duration [1.5s], collections [1]/[1.6s], total [1.5s]/[14.1s], memory [21.3gb]-\>[22.1gb]/[31.8gb], all\_pools {[young] [26mb]-\>[28.4mb]/[865.3mb]}{[survivor] [108.1mb]-\>[108.1mb]/[108.1mb]}{[old] [21.2gb]-\>[22gb]/[30.9gb]} [2017-11-10 08:26:41,217][WARN][monitor.jvm] [10.0.1.95] [gc][young][1077][35] duration [1.4s], collections [1]/[1.5s], total [1.4s]/[15.6s], memory [22.1gb]-\>[23gb]/[31.8gb], all\_pools {[young] [28.4mb]-\>[27.4mb]/[865.3mb]}{[survivor] [108.1mb]-\>[108.1mb]/[108.1mb]}{[old] [22gb]-\>[22.8gb]/[30.9gb]} [2017-11-10 08:26:42,990][WARN][monitor.jvm] [10.0.1.95] [gc][young][1078][36] duration [1.6s], collections [1]/[1.7s], total [1.6s]/[17.2s], memory [23gb]-\>[23.8gb]/[31.8gb], all\_pools {[young] [27.4mb]-\>[17.4mb]/[865.3mb]}{[survivor] [108.1mb]-\>[108.1mb]/[108.1mb]}{[old] [22.8gb]-\>[23.7gb]/[30.9gb]} [2017-11-10 08:26:45,167][WARN][monitor.jvm] [10.0.1.95] [gc][young][1079][37] duration [2s], collections [1]/[2.1s], total [2s]/[19.3s], memory [23.8gb]-\>[24.7gb]/[31.8gb], all\_pools {[young] [17.4mb]-\>[34.6mb]/[865.3mb]}{[survivor] [108.1mb]-\>[108.1mb]/[108.1mb]}{[old] [23.7gb]-\>[24.5gb]/[30.9gb]} [2017-11-10 08:26:47,178][WARN][monitor.jvm] [10.0.1.95] [gc][young][1080][38] duration [1.9s], collections [1]/[2s], total [1.9s]/[21.2s], memory [24.7gb]-\>[25.5gb]/[31.8gb], all\_pools {[young] [34.6mb]-\>[35.2mb]/[865.3mb]}{[survivor] [108.1mb]-\>[108.1mb]/[108.1mb]}{[old] [24.5gb]-\>[25.4gb]/[30.9gb]} [2017-11-10 08:26:48,714][WARN][monitor.jvm] [10.0.1.95] [gc][young][1081][39] duration [1.4s], collections [1]/[1.5s], total [1.4s]/[22.7s], memory [25.5gb]-\>[26.3gb]/[31.8gb], all\_pools {[young] [35.2mb]-\>[25.9mb]/[865.3mb]}{[survivor] [108.1mb]-\>[108.1mb]/[108.1mb]}{[old] [25.4gb]-\>[26.2gb]/[30.9gb]} [2017-11-10 08:26:50,547][WARN][monitor.jvm] [10.0.1.95] [gc][young][1082][40] duration [1.7s], collections [1]/[1.8s], total [1.7s]/[24.4s], memory [26.3gb]-\>[27.2gb]/[31.8gb], all\_pools {[young] [25.9mb]-\>[27.5mb]/[865.3mb]}{[survivor] [108.1mb]-\>[108.1mb]/[108.1mb]}{[old] [26.2gb]-\>[27gb]/[30.9gb]} [2017-11-10 08:26:52,066][WARN][monitor.jvm] [10.0.1.95] [gc][young][1083][41] duration [1.4s], collections [1]/[1.5s], total [1.4s]/[25.9s], memory [27.2gb]-\>[28gb]/[31.8gb], all\_pools {[young] [27.5mb]-\>[26.1mb]/[865.3mb]}{[survivor] [108.1mb]-\>[108.1mb]/[108.1mb]}{[old] [27gb]-\>[27.9gb]/[30.9gb]} [2017-11-10 08:26:53,067][INFO][monitor.jvm] [10.0.1.95] [gc][young][1084][42] duration [879ms], collections [1]/[1s], total [879ms]/[26.7s], memory [28gb]-\>[29.3gb]/[31.8gb], all\_pools {[young] [26.1mb]-\>[472.2mb]/[865.3mb]}{[survivor] [108.1mb]-\>[108.1mb]/[108.1mb]}{[old] [27.9gb]-\>[28.7gb]/[30.9gb]} [2017-11-10 08:27:04,654][WARN][monitor.jvm] [10.0.1.95] [gc][old][1086][2] duration [10.3s], collections [1]/[10.4s], total [10.3s]/[10.4s], memory [30.5gb]-\>[31.3gb]/[31.8gb], all\_pools {[young] [25.9mb]-\>[450.2mb]/[865.3mb]}{[survivor] [108.1mb]-\>[0b]/[108.1mb]}{[old] [30.4gb]-\>[30.9gb]/[30.9gb]} [2017-11-10 08:27:12,423][INFO][monitor.jvm] [10.0.1.95] [gc][old][1087][3] duration [7.7s], collections [1]/[7.7s], total [7.7s]/[18.1s], memory [31.3gb]-\>[31.8gb]/[31.8gb], all\_pools {[young] [450.2mb]-\>[865.3mb]/[865.3mb]}{[survivor] [0b]-\>[105.6mb]/[108.1mb]}{[old] [30.9gb]-\>[30.9gb]/[30.9gb]} Question: how can we stop or minimise GC "STOP-THE-WORLD" issue? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 8, 2017, 12:06pm UTC](https://discuss.elastic.co/t/aggregation-query-making-elastic-datanode-offline-due-to-long-gc/107179/2 "2017-12-08T12:06:31Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.