# Please, help me tune java GC parameters **URL:** \ **Category:** Elasticsearch\ **Created:** [November 23, 2016, 4:56am UTC](https://discuss.elastic.co/t/please-help-me-tune-java-gc-parameters/66940 "2016-11-23T04:56:43Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Brad\_Jungsu\_Heo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brad_jungsu_heo/32/13135_2.png) [@Brad\_Jungsu\_Heo](https://discuss.elastic.co/u/Brad_Jungsu_Heo)\ **Post date:** [November 23, 2016, 4:56am UTC](https://discuss.elastic.co/t/please-help-me-tune-java-gc-parameters/66940/1 "2016-11-23T04:56:43Z")
Hi. I have a trouble with long java gc Stop-The-World during heavy bulk load. my environment - ES Version: 2.3.2 - Heap Size: 20GB - Total Ram: 64GB - Java Version: 1.8.0\_60 Java HotSpot(TM) 64-Bit - OS: CentOS 7.2 1. gc log with default parameters * * * At the initial deployment of my cluster, I had experienced stw like this: ```auto [date][WARN][monitor.jvm] [hostname] [gc][old][811366][210] duration [10s], collections [1]/[11.3s], total [10s]/[1.1m], memory [14.5gb]->[2.7gb]/[19.8gb], all_pools {[young] [1gb]->[25.9mb]/[1.4gb]} {[survivor] [184.5mb]->[0b]/[191.3mb]} {[old] [13.2gb]->[2.7gb]/[18.1gb]} ``` 1. Parameters tuned * * * So, I've tuned gc parameters. young and survivor area increased to prevent from new object are moved to old generation. ```auto -XX:MaxTenuringThreshold=15 -XX:NewRatio=6 -XX:SurvivorRatio=3 -XX:-UseAdaptiveSizePolicy ``` 1. after several months later * * * Full gc happens after tuning but less frequently. It was fine for several months. But earlier of this month, I got longer stw than initial gc parameters. ```auto [date][WARN][monitor.jvm] [hostname] [gc][old][2894376][121395] duration [28.7s], collections [1]/[30s], total [28.7s]/[4.4h], memory [15.8gb]->[14.6gb]/[19.4gb], all_pools {[young] [498.1mb]->[83.3mb]/[1.7gb]} {[survivor] [394.6mb]->[0b]/[585.1mb]} {[old] [15gb]->[14.5gb]/[17.1gb]} ``` After gc, only 0.5gb of old generation was freed. this long stw happens again every 5~7 days. The only thing I can do is Rolling Restart ☹ It would be appreciated it if anyone can suggest gc parameters or help me out! Thanks in advance ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 23, 2016, 8:55am UTC](https://discuss.elastic.co/t/please-help-me-tune-java-gc-parameters/66940/2 "2016-11-23T08:55:52Z")
We do not recommend tuning java parameters. If you are constantly hitting GC then your cluster is overloaded, and any tweaking you do is of diminishing returns. ---
**Author:** ![Brad\_Jungsu\_Heo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brad_jungsu_heo/32/13135_2.png) [@Brad\_Jungsu\_Heo](https://discuss.elastic.co/u/Brad_Jungsu_Heo)\ **Post date:** [November 23, 2016, 9:29am UTC](https://discuss.elastic.co/t/please-help-me-tune-java-gc-parameters/66940/3 "2016-11-23T09:29:21Z")
@warkolm Thank you for your reply! After reading your reply, I've read [Don’t Touch These Settings!](https://www.elastic.co/guide/en/elasticsearch/guide/current/_don_8217_t_touch_these_settings.html). There is `"Do not change the default garbage collector!"` but there is no "We do not recommend tuning java parameters." I'd like to change CMS gc's parameters. Is this not recommended? and Could I read some related articles? Thank you. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 23, 2016, 9:35am UTC](https://discuss.elastic.co/t/please-help-me-tune-java-gc-parameters/66940/4 "2016-11-23T09:35:14Z")
What does you Elasticsearch config look like? Do you have scripting enabled? ---
**Author:** ![Ravi\_Shanker\_Reddy](https://avatars.discourse-cdn.com/v4/letter/r/a5b964/32.png) [@Ravi\_Shanker\_Reddy](https://discuss.elastic.co/u/Ravi_Shanker_Reddy)\ **Post date:** [November 23, 2016, 9:35am UTC](https://discuss.elastic.co/t/please-help-me-tune-java-gc-parameters/66940/5 "2016-11-23T09:35:53Z")
[https://www.elastic.co/guide/en/elasticsearch/guide/current/\_java\_virtual\_machine.html](https://www.elastic.co/guide/en/elasticsearch/guide/current/_java_virtual_machine.html) Just FYI. In this they mentioned Clearly not to tweak any JVM settings. ---
**Author:** ![Brad\_Jungsu\_Heo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brad_jungsu_heo/32/13135_2.png) [@Brad\_Jungsu\_Heo](https://discuss.elastic.co/u/Brad_Jungsu_Heo)\ **Post date:** [November 23, 2016, 9:40am UTC](https://discuss.elastic.co/t/please-help-me-tune-java-gc-parameters/66940/6 "2016-11-23T09:40:09Z")
@Christian_Dahlqvist Yes. scripting is enabled and approximately 30m documents are updated every day. I'm wondering scripting and updates are related with full gc. Thanks. ---
**Author:** ![Brad\_Jungsu\_Heo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brad_jungsu_heo/32/13135_2.png) [@Brad\_Jungsu\_Heo](https://discuss.elastic.co/u/Brad_Jungsu_Heo)\ **Post date:** [November 23, 2016, 9:41am UTC](https://discuss.elastic.co/t/please-help-me-tune-java-gc-parameters/66940/7 "2016-11-23T09:41:01Z")
@Ravi_Shanker_Reddy Oh. I got it. I'll read it carefully. Thank you. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 23, 2016, 9:47am UTC](https://discuss.elastic.co/t/please-help-me-tune-java-gc-parameters/66940/8 "2016-11-23T09:47:45Z")
According to [this post](https://discuss.elastic.co/t/elastic-search-using-a-lot-of-memory-gc-thrashing/48695/2) there may be a memory leak in Groovy, which might be causing the issues you are seeing. ---
**Author:** ![Brad\_Jungsu\_Heo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brad_jungsu_heo/32/13135_2.png) [@Brad\_Jungsu\_Heo](https://discuss.elastic.co/u/Brad_Jungsu_Heo)\ **Post date:** [November 23, 2016, 9:54am UTC](https://discuss.elastic.co/t/please-help-me-tune-java-gc-parameters/66940/9 "2016-11-23T09:54:32Z")
@Christian_Dahlqvist Great! I didn't know that. Thank you so much. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 23, 2016, 9:56am UTC](https://discuss.elastic.co/t/please-help-me-tune-java-gc-parameters/66940/10 "2016-11-23T09:56:38Z")
It might be worthwhile looking into migrating to Elasticsearch 5.0 and [the new Painless scripting language](https://www.elastic.co/guide/en/elasticsearch/reference/5.0/breaking_50_scripting.html#_switched_default_language_from_groovy_to_painless). ---
**Author:** ![Brad\_Jungsu\_Heo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brad_jungsu_heo/32/13135_2.png) [@Brad\_Jungsu\_Heo](https://discuss.elastic.co/u/Brad_Jungsu_Heo)\ **Post date:** [November 24, 2016, 4:33am UTC](https://discuss.elastic.co/t/please-help-me-tune-java-gc-parameters/66940/11 "2016-11-24T04:33:31Z")
@Christian_Dahlqvist Thank you for your consideration. But I can't use ES 5.0 at this moment because I use CDH 5.8 (which uses jdk 1.7), es-hadoop, and Spark (ES 5.0's es-hadoop only supports java 1.8) and with regars to memory leaks of groovy script, I execute a same script for every request, like this: ```auto val upsertScript = """ ... if (ctx._source.containsKey("f1")) { ctx._source.f1 += param1; } else { ctx._source.f1 = param1; } ... """ conf.set("es.update.script", upsertScript); conf.set("es.update.script.params", "param1:v1,param2:v2,param3:v3") ``` So, I think my deployment does not relate to memory-leak. Anyway, I appreciate you reply! Thanks. ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [November 24, 2016, 6:20pm UTC](https://discuss.elastic.co/t/please-help-me-tune-java-gc-parameters/66940/12 "2016-11-24T18:20:49Z")
The memory leak was fixed by the Groovy team [https://issues.apache.org/jira/browse/GROOVY-7683](https://issues.apache.org/jira/browse/GROOVY-7683) so you should watch closely when Groovy 2.4.8 will be released. Then, replace existing groovy jar in ES to the new 2.4.8 groovy jar. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 22, 2016, 6:20pm UTC](https://discuss.elastic.co/t/please-help-me-tune-java-gc-parameters/66940/13 "2016-12-22T18:20:54Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.