# JRE 1.7.0\_11 / ES 1.0.1 - GC not collecting old gen / Memory Leak? **URL:** \ **Category:** Elasticsearch\ **Created:** [October 20, 2014, 1:27pm UTC](https://discuss.elastic.co/t/jre-1-7-0-11-es-1-0-1-gc-not-collecting-old-gen-memory-leak/20339 "2014-10-20T13:27:35Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Gavin\_Seng](https://avatars.discourse-cdn.com/v4/letter/g/e56c9b/32.png) [@Gavin\_Seng](https://discuss.elastic.co/u/Gavin_Seng)\ **Post date:** [October 20, 2014, 1:27pm UTC](https://discuss.elastic.co/t/jre-1-7-0-11-es-1-0-1-gc-not-collecting-old-gen-memory-leak/20339/1 "2014-10-20T13:27:35Z")
### JRE 1.7.0\_11 / ES 1.0.1 - GC not collecting old gen / Memory Leak? Hi, We're seeing issues where GC collects less and less memory over time leading to the need to restart our nodes. The following is our setup and what we've tried. Please tell me if anything is lacking and I'll be glad to provide more details. Also appreciate any advice on how we can improve our configurations. Thank you for any help! Gavin ### Cluster Setup - Tribes that link to 2 clusters - Cluster 1 - Cluster 2 Interestingly, we're actually having problems on Cluster 1's hot nodes even though it indexes less. It suggests that this is a problem with searching because Cluster 1 is searched on a lot more. ### Machine settings (hot node) - java - 128gb ram - 8 cores, 32 cpus - ssds (raid 0) ### JVM settings ```auto java -Xms96g -Xmx96g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintClassHistogram -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/log/elasticsearch/gc.log -XX:+HeapDumpOnOutOfMemoryError -verbose:gc -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -Xloggc:[...] -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=[...] -Dcom.sun.management.jmxremote.ssl=[...] -Dcom.sun.management.jmxremote.authenticate=[...] -Dcom.sun.management.jmxremote.port=[...] -Delasticsearch -Des.pidfile=[...] -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.0.1.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=[...] -Des.default.path.data=[...] -Des.default.path.work=[...] -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch ``` ## Key elasticsearch.yml settings - threadpool.bulk.type: fixed - threadpool.bulk.queue\_size: 1000 - indices.memory.index\_buffer\_size: 30% - index.translog.flush\_threshold\_ops: 50000 - indices.fielddata.cache.size: 30% ### Search Load (Cluster 1) - Mainly Kibana3 (queries ES with daily alias that expands to 24 hourly indices) - Jenkins jobs that constantly run and do many faceting/aggregations for the last hour's of data ### Things we've tried (unsuccesfully) - GC settings - Limit filter/field cache - Optimizing index to 1 segment on the 3rd hour - Limit JVM to 32 gb ram - Limit JVM to 65 gb ram - Read 90.5/7 OOM errors-- memory leak or GC problems? [https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/memory$20leak/elasticsearch/\_Zve60xOh\_E/N13tlXgkUAwJ](https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/memory%2420leak/elasticsearch/_Zve60xOh_E/N13tlXgkUAwJ) - But we're not using term filters ### 32 GB Heap [https://lh3.googleusercontent.com/-T6uUeqSFhns/VEUMsYWwukI/AAAAAAAABm0/eayuSevxWNY/s1600/es\_32gb.png](https://lh3.googleusercontent.com/-T6uUeqSFhns/VEUMsYWwukI/AAAAAAAABm0/eayuSevxWNY/s1600/es_32gb.png) ### 65 GB Heap [https://lh4.googleusercontent.com/-C9ScRI9pO2A/VEUM6uxcJ-I/AAAAAAAABm8/iGqqKemt4aw/s1600/es\_65gb.png](https://lh4.googleusercontent.com/-C9ScRI9pO2A/VEUM6uxcJ-I/AAAAAAAABm8/iGqqKemt4aw/s1600/es_65gb.png) ### 65 GB Heap with changed young/old ratio [https://lh4.googleusercontent.com/-Ugzr4PQv\_uE/VEUNGy-zguI/AAAAAAAABnE/FbnhnVHQQ20/s1600/es\_65gb\_yo.png](https://lh4.googleusercontent.com/-Ugzr4PQv_uE/VEUNGy-zguI/AAAAAAAABnE/FbnhnVHQQ20/s1600/es_65gb_yo.png) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/33c2b354-975d-4340-ad77-4d675e577339%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/33c2b354-975d-4340-ad77-4d675e577339%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [October 20, 2014, 7:57pm UTC](https://discuss.elastic.co/t/jre-1-7-0-11-es-1-0-1-gc-not-collecting-old-gen-memory-leak/20339/2 "2014-10-20T19:57:48Z")
Hi Gavin, You might be hit by the following Guava bug: [Internal: Filter cache size limit not honored for 32GB or over · Issue #6268 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/6268). It was fixed in Elasticsearch 1.1.3/1.2.1/1.3.0 On Mon, Oct 20, 2014 at 3:27 PM, Gavin Seng [seng.gavin@gmail.com](mailto:seng.gavin@gmail.com) wrote: > ### JRE 1.7.0\_11 / ES 1.0.1 - GC not collecting old gen / Memory Leak? > > Hi, > > We're seeing issues where GC collects less and less memory over time > leading to the need to restart our nodes. > > The following is our setup and what we've tried. Please tell me if > anything is lacking and I'll be glad to provide more details. > > Also appreciate any advice on how we can improve our configurations. > > Thank you for any help! > > Gavin > > ### Cluster Setup > > - Tribes that link to 2 clusters > > - Cluster 1 > > - Cluster 2 > > Interestingly, we're actually having problems on Cluster 1's hot nodes > even though it indexes less. > > It suggests that this is a problem with searching because Cluster 1 is > searched on a lot more. > > ### Machine settings (hot node) > > - java > > - 128gb ram > > - 8 cores, 32 cpus > > - ssds (raid 0) > > ### JVM settings > > ```auto > > java > > -Xms96g -Xmx96g -Xss256k > > -Djava.awt.headless=true > > -XX:+UseParNewGC -XX:+UseConcMarkSweepGC > -XX:CMSInitiatingOccupancyFraction=75 > > -XX:+UseCMSInitiatingOccupancyOnly > > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintClassHistogram > -XX:+PrintTenuringDistribution > > -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/log/elasticsearch/gc.log > -XX:+HeapDumpOnOutOfMemoryError > > -verbose:gc -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation > -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M > > -Xloggc:[...] > > -Dcom.sun.management.jmxremote > -Dcom.sun.management.jmxremote.local.only=[...] > > -Dcom.sun.management.jmxremote.ssl=[...] > -Dcom.sun.management.jmxremote.authenticate=[...] > > -Dcom.sun.management.jmxremote.port=[...] > > -Delasticsearch -Des.pidfile=[...] > > -Des.path.home=/usr/share/elasticsearch -cp > :/usr/share/elasticsearch/lib/elasticsearch-1.0.1.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* > > -Des.default.path.home=/usr/share/elasticsearch > > -Des.default.path.logs=[...] > > -Des.default.path.data=[...] > > -Des.default.path.work=[...] > > -Des.default.path.conf=/etc/elasticsearch > org.elasticsearch.bootstrap.Elasticsearch > > ``` > > ## Key elasticsearch.yml settings > > - threadpool.bulk.type: fixed > > - threadpool.bulk.queue\_size: 1000 > > - indices.memory.index\_buffer\_size: 30% > > - index.translog.flush\_threshold\_ops: 50000 > > - indices.fielddata.cache.size: 30% > > ### Search Load (Cluster 1) > > - Mainly Kibana3 (queries ES with daily alias that expands to 24 hourly > indices) > > - Jenkins jobs that constantly run and do many faceting/aggregations for > the last hour's of data > > ### Things we've tried (unsuccesfully) > > - GC settings > > - Limit filter/field cache > > - Optimizing index to 1 segment on the 3rd hour > > - Limit JVM to 32 gb ram > > - Limit JVM to 65 gb ram > > - Read 90.5/7 OOM errors-- memory leak or GC problems? > > [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/memory$20leak/elasticsearch/_Zve60xOh_E/N13tlXgkUAwJ) > > - But we're not using term filters > > ### 32 GB Heap > > [https://lh3.googleusercontent.com/-T6uUeqSFhns/VEUMsYWwukI/AAAAAAAABm0/eayuSevxWNY/s1600/es\_32gb.png](https://lh3.googleusercontent.com/-T6uUeqSFhns/VEUMsYWwukI/AAAAAAAABm0/eayuSevxWNY/s1600/es_32gb.png) > > ### 65 GB Heap > > [https://lh4.googleusercontent.com/-C9ScRI9pO2A/VEUM6uxcJ-I/AAAAAAAABm8/iGqqKemt4aw/s1600/es\_65gb.png](https://lh4.googleusercontent.com/-C9ScRI9pO2A/VEUM6uxcJ-I/AAAAAAAABm8/iGqqKemt4aw/s1600/es_65gb.png) > > ### 65 GB Heap with changed young/old ratio > > [https://lh4.googleusercontent.com/-Ugzr4PQv\_uE/VEUNGy-zguI/AAAAAAAABnE/FbnhnVHQQ20/s1600/es\_65gb\_yo.png](https://lh4.googleusercontent.com/-Ugzr4PQv_uE/VEUNGy-zguI/AAAAAAAABnE/FbnhnVHQQ20/s1600/es_65gb_yo.png) > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/33c2b354-975d-4340-ad77-4d675e577339%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/33c2b354-975d-4340-ad77-4d675e577339%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/33c2b354-975d-4340-ad77-4d675e577339%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/33c2b354-975d-4340-ad77-4d675e577339%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6MOySTUytZLN4%2BQ6iWuysXpb7BmzZMxGjzqBqEyWR%2BpQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6MOySTUytZLN4%2BQ6iWuysXpb7BmzZMxGjzqBqEyWR%2BpQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Gavin\_Seng](https://avatars.discourse-cdn.com/v4/letter/g/e56c9b/32.png) [@Gavin\_Seng](https://discuss.elastic.co/u/Gavin_Seng)\ **Post date:** [October 21, 2014, 1:19am UTC](https://discuss.elastic.co/t/jre-1-7-0-11-es-1-0-1-gc-not-collecting-old-gen-memory-leak/20339/3 "2014-10-21T01:19:25Z")
Please post all updates here (has pictures and better formatting): [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups=#!topic/elasticsearch/VxkosQuKzaA) Thanks Adrien, I've cross-posted your reply in the other post. On Monday, October 20, 2014 3:57:56 PM UTC-4, Adrien Grand wrote: > Hi Gavin, > > You might be hit by the following Guava bug: > [Internal: Filter cache size limit not honored for 32GB or over · Issue #6268 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/6268). It was fixed > in Elasticsearch 1.1.3/1.2.1/1.3.0 > > On Mon, Oct 20, 2014 at 3:27 PM, Gavin Seng \<[seng....@gmail.com](mailto:seng....@gmail.com) > \\> wrote: > > > ### JRE 1.7.0\_11 / ES 1.0.1 - GC not collecting old gen / Memory Leak? > > > > Hi, > > > > We're seeing issues where GC collects less and less memory over time > > leading to the need to restart our nodes. > > > > The following is our setup and what we've tried. Please tell me if > > anything is lacking and I'll be glad to provide more details. > > > > Also appreciate any advice on how we can improve our configurations. > > > > Thank you for any help! > > > > Gavin > > > > ### Cluster Setup > > > > - Tribes that link to 2 clusters > > > > - Cluster 1 > > > > - Cluster 2 > > > > Interestingly, we're actually having problems on Cluster 1's hot nodes > > even though it indexes less. > > > > It suggests that this is a problem with searching because Cluster 1 is > > searched on a lot more. > > > > ### Machine settings (hot node) > > > > - java > > > > - 128gb ram > > > > - 8 cores, 32 cpus > > > > - ssds (raid 0) > > > > ### JVM settings > > > > ```auto > > > > > > java > > > > > > -Xms96g -Xmx96g -Xss256k > > > > > > -Djava.awt.headless=true > > > > > > -XX:+UseParNewGC -XX:+UseConcMarkSweepGC > > -XX:CMSInitiatingOccupancyFraction=75 > > > > > > -XX:+UseCMSInitiatingOccupancyOnly > > > > > > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintClassHistogram > > -XX:+PrintTenuringDistribution > > > > -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/log/elasticsearch/gc.log > > -XX:+HeapDumpOnOutOfMemoryError > > > > -verbose:gc -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation > > -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M > > > > -Xloggc:[...] > > > > > > -Dcom.sun.management.jmxremote > > -Dcom.sun.management.jmxremote.local.only=[...] > > > > > > -Dcom.sun.management.jmxremote.ssl=[...] > > -Dcom.sun.management.jmxremote.authenticate=[...] > > > > > > -Dcom.sun.management.jmxremote.port=[...] > > > > > > -Delasticsearch -Des.pidfile=[...] > > > > > > -Des.path.home=/usr/share/elasticsearch -cp > > :/usr/share/elasticsearch/lib/elasticsearch-1.0.1.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* > > > > -Des.default.path.home=/usr/share/elasticsearch > > > > > > -Des.default.path.logs=[...] > > > > > > -Des.default.path.data=[...] > > > > > > -Des.default.path.work=[...] > > > > > > -Des.default.path.conf=/etc/elasticsearch > > org.elasticsearch.bootstrap.Elasticsearch > > > > > > > > ``` > > > > ## Key elasticsearch.yml settings > > > > - threadpool.bulk.type: fixed > > > > - threadpool.bulk.queue\_size: 1000 > > > > - indices.memory.index\_buffer\_size: 30% > > > > - index.translog.flush\_threshold\_ops: 50000 > > > > - indices.fielddata.cache.size: 30% > > > > ### Search Load (Cluster 1) > > > > - Mainly Kibana3 (queries ES with daily alias that expands to 24 hourly > > indices) > > > > - Jenkins jobs that constantly run and do many faceting/aggregations for > > the last hour's of data > > > > ### Things we've tried (unsuccesfully) > > > > - GC settings > > > > - Limit filter/field cache > > > > - Optimizing index to 1 segment on the 3rd hour > > > > - Limit JVM to 32 gb ram > > > > - Limit JVM to 65 gb ram > > > > - Read 90.5/7 OOM errors-- memory leak or GC problems? > > > > [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/memory$20leak/elasticsearch/_Zve60xOh_E/N13tlXgkUAwJ) > > > > - But we're not using term filters > > > > ### 32 GB Heap > > > > [https://lh3.googleusercontent.com/-T6uUeqSFhns/VEUMsYWwukI/AAAAAAAABm0/eayuSevxWNY/s1600/es\_32gb.png](https://lh3.googleusercontent.com/-T6uUeqSFhns/VEUMsYWwukI/AAAAAAAABm0/eayuSevxWNY/s1600/es_32gb.png) > > > > ### 65 GB Heap > > > > [https://lh4.googleusercontent.com/-C9ScRI9pO2A/VEUM6uxcJ-I/AAAAAAAABm8/iGqqKemt4aw/s1600/es\_65gb.png](https://lh4.googleusercontent.com/-C9ScRI9pO2A/VEUM6uxcJ-I/AAAAAAAABm8/iGqqKemt4aw/s1600/es_65gb.png) > > > > ### 65 GB Heap with changed young/old ratio > > > > [https://lh4.googleusercontent.com/-Ugzr4PQv\_uE/VEUNGy-zguI/AAAAAAAABnE/FbnhnVHQQ20/s1600/es\_65gb\_yo.png](https://lh4.googleusercontent.com/-Ugzr4PQv_uE/VEUNGy-zguI/AAAAAAAABnE/FbnhnVHQQ20/s1600/es_65gb_yo.png) > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/33c2b354-975d-4340-ad77-4d675e577339%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/33c2b354-975d-4340-ad77-4d675e577339%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/33c2b354-975d-4340-ad77-4d675e577339%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/33c2b354-975d-4340-ad77-4d675e577339%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5f57c616-3f67-4157-a28b-dcf4fc0478b0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5f57c616-3f67-4157-a28b-dcf4fc0478b0%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:54am UTC](https://discuss.elastic.co/t/jre-1-7-0-11-es-1-0-1-gc-not-collecting-old-gen-memory-leak/20339/4 "2017-07-06T00:54:51Z")