# Endless GC. Need update about GC best practices for operation **URL:** \ **Category:** Elasticsearch\ **Created:** [February 22, 2018, 4:09pm UTC](https://discuss.elastic.co/t/endless-gc-need-update-about-gc-best-practices-for-operation/121085 "2018-02-22T16:09:10Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![DidierB](https://avatars.discourse-cdn.com/v4/letter/d/ed8c4c/32.png) [@DidierB](https://discuss.elastic.co/u/DidierB)\ **Post date:** [February 22, 2018, 4:09pm UTC](https://discuss.elastic.co/t/endless-gc-need-update-about-gc-best-practices-for-operation/121085/1 "2018-02-22T16:09:10Z")
Hello, I have a 12 nodes 6.0.0 elasticsearch cluster. It ingest 10k.ev/sec. Time to time, I think because of some Kibana request, the cluster stops to answer to kibana and even to local curls. The logs are pretty clear: ``` [2018-02-22T16:36:43,573][WARN][o.e.m.j.JvmGcMonitorService] [serverA] [gc][old][1748945][386] duration [33.3s], collections [1]/[33.6s], total [33.3s]/[31m], memory [29.7gb]->[29.7gb]/[29.7gb], all_pools {[young] [1.8gb]->[1.8gb]/[1.8gb]}{[survivor] [181.2mb]->[185.3mb]/[232.9mb]}{[old] [27.7gb]->[27.7gb]/[27.7gb]} [2018-02-22T16:36:43,581][WARN][o.e.m.j.JvmGcMonitorService] [serverA] [gc][1748945] overhead, spent [33.3s] collecting in the last [33.6s] ``` Is garbage collecting for 33.3s in the last 33.6s a bad thing? 🙂 Some extract of the jvm options: ``` -Xms30g -Xmx30g ## GC configuration -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Xss1m ``` After this endless GC, nodes do not answer to ping and the cluster doesn't answer and need node restarts. Any advice about how to make this not happen would be welcome, or how to recover without restarts, would be welcome. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 22, 2018, 4:22pm UTC](https://discuss.elastic.co/t/endless-gc-need-update-about-gc-best-practices-for-operation/121085/2 "2018-02-22T16:22:15Z")
Do you have monitoring installed? Do you have a graph showing how heap usage varies over time? ---
**Author:** ![DidierB](https://avatars.discourse-cdn.com/v4/letter/d/ed8c4c/32.png) [@DidierB](https://discuss.elastic.co/u/DidierB)\ **Post date:** [February 22, 2018, 4:35pm UTC](https://discuss.elastic.co/t/endless-gc-need-update-about-gc-best-practices-for-operation/121085/3 "2018-02-22T16:35:15Z")
Thanks for you answer. I don't have access to this kind of monitoring but I can tell that the cluster can operate for weeks before this happen. Every time I have this problem, on a smaller cluster, was because of a too-big request. I can't prevent the user to do big requests, I just want them to fail when there is catastrophic consequences on the cluster. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 22, 2018, 4:39pm UTC](https://discuss.elastic.co/t/endless-gc-need-update-about-gc-best-practices-for-operation/121085/4 "2018-02-22T16:39:55Z")
It would be useful to see how heap is being used and how much headroom you have for large queries. The [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/cluster-stats.html) could provide some of this information. Do you have any non-standard configuration around memory and/or [circuit breakers](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/circuit-breaker.html)? ---
**Author:** ![DidierB](https://avatars.discourse-cdn.com/v4/letter/d/ed8c4c/32.png) [@DidierB](https://discuss.elastic.co/u/DidierB)\ **Post date:** [February 22, 2018, 4:56pm UTC](https://discuss.elastic.co/t/endless-gc-need-update-about-gc-best-practices-for-operation/121085/5 "2018-02-22T16:56:23Z")
I restarted the cluster so I don't know if the cluster stat will help you. I however plane to collect regular statistics. I didn't know about the [circuit breakers](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/circuit-breaker.html) so I guess there is none. I'll study this page immediately. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 22, 2018, 5:00pm UTC](https://discuss.elastic.co/t/endless-gc-need-update-about-gc-best-practices-for-operation/121085/6 "2018-02-22T17:00:05Z")
Cluster stats will give an indication about baseline heap usage, so might give some insight. Not as good as a graph though... ---
**Author:** ![DidierB](https://avatars.discourse-cdn.com/v4/letter/d/ed8c4c/32.png) [@DidierB](https://discuss.elastic.co/u/DidierB)\ **Post date:** [February 23, 2018, 10:40am UTC](https://discuss.elastic.co/t/endless-gc-need-update-about-gc-best-practices-for-operation/121085/7 "2018-02-23T10:40:47Z")
Thanks to your advice, I put a total circuit breaker of 45% of Heap. I'll test in a few hour if it works. Which brings me to the next point. I need more heap space. I understand the java pointer compression and my XMX is set to 30GB. However my nodes have 256GB of memory. Is it safe to put XMX to 80GB or so? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 23, 2018, 10:48am UTC](https://discuss.elastic.co/t/endless-gc-need-update-about-gc-best-practices-for-operation/121085/8 "2018-02-23T10:48:29Z")
Going beyond 30GB is as far as I can recall still not receommended. What you can do in order to optimise the heap space on large hosts is however to run multiple nodes on it. A 256GB host should be able to handle 4 nodes , each with a 30GB heap. ---
**Author:** ![DidierB](https://avatars.discourse-cdn.com/v4/letter/d/ed8c4c/32.png) [@DidierB](https://discuss.elastic.co/u/DidierB)\ **Post date:** [February 23, 2018, 11:37am UTC](https://discuss.elastic.co/t/endless-gc-need-update-about-gc-best-practices-for-operation/121085/9 "2018-02-23T11:37:46Z")
Unfortunately I'm not able to doing this kind of modification on the production cluster. Plus, to be efficient I would have to double the number of shards for each indexes (and this halve their size), if not the same request would saturate the JVM again, isn't it? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 23, 2018, 11:37am UTC](https://discuss.elastic.co/t/endless-gc-need-update-about-gc-best-practices-for-operation/121085/10 "2018-03-23T11:37:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.