# Garbage collector issues **URL:** \ **Category:** Elasticsearch\ **Created:** [November 14, 2013, 2:06pm UTC](https://discuss.elastic.co/t/garbage-collector-issues/14407 "2013-11-14T14:06:56Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![vincent\_miszczak](https://avatars.discourse-cdn.com/v4/letter/v/b77776/32.png) [@vincent\_miszczak](https://discuss.elastic.co/u/vincent_miszczak)\ **Post date:** [November 14, 2013, 2:06pm UTC](https://discuss.elastic.co/t/garbage-collector-issues/14407/1 "2013-11-14T14:06:56Z")
Hi, We are running Elasticsearch 0.90.2 on Debian 7.0/OpenJDK7u3 (2 nodes cluster). From time to time, Elasticsearch stop responding and the issue looks related to the Garbage Collector. Here are the information we have collected when problems occur : -The search threadpool hits the concurrent active items limit and the queue limit (default values, ie 36 threads and 1000 slots in the queue). -We have high rate of slow queries (\>8 seconds) -The garbage collector logs long passes (around 6 seconds) -Clients get Rejected exceptions All of this happens for several minutes (\> 10) from time to time Then everything get back to normal. Logs are attached. The values we have : System total memory : 6GB ES\_HEAP\_SIZE=3g We are almost sure this issue comes from long GC run. We are planning to change the GC for G1 (after upgrading to Java 7u25, because this GC requires Java 7u4), but I've seen in this group one thread saying it crashes:/ What can we do to prevent this behavior and run ES smoothly ? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [November 14, 2013, 5:42pm UTC](https://discuss.elastic.co/t/garbage-collector-issues/14407/2 "2013-11-14T17:42:45Z")
Hello Vincent, If the search threadpool hits the limit maybe you have too many concurrent searches. If that's the case, you'll probably have to just add nodes and/or increase the number of replicas. Or, you can look at making your queries faster, if that's possible. G1 may help, I would try it and see how it goes. Last but not least, I would look at what is consuming memory. Is it field cache? Is it filter cache? I think nodes stats can tell you that, and you could turn a few knobs there and limit memory usage: > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... You may also want to try out our SPM for Elasticsearch. It will show you all sorts of metrics, from Garbage Collector and pool sizes to cache sizes. I assume it would be very helpful in this particular case: > **[Elasticsearch - Sematext Documentation](https://sematext.com/docs/integration/elasticsearch-integration/)** > > Collect and monitor key Elasticsearch metrics such as request latency, indexing rate, and segment merges with built-in anomaly detection, threshold, and heartbeat alerts. Send notifications to email and various chatops messaging services, correlate... ## Best regards, Radu Performance Monitoring \* Log Analytics \* Search Analytics Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) On Thu, Nov 14, 2013 at 6:06 AM, vincent miszczak \< [vincent.miszczak@gmail.com](mailto:vincent.miszczak@gmail.com)\> wrote: > Hi, > > We are running Elasticsearch 0.90.2 on Debian 7.0/OpenJDK7u3 (2 nodes > cluster). > From time to time, Elasticsearch stop responding and the issue looks > related to the Garbage Collector. > > Here are the information we have collected when problems occur : > -The search threadpool hits the concurrent active items limit and the > queue limit (default values, ie 36 threads and 1000 slots in the queue). > -We have high rate of slow queries (\>8 seconds) > -The garbage collector logs long passes (around 6 seconds) > -Clients get Rejected exceptions > All of this happens for several minutes (\> 10) from time to time > > Then everything get back to normal. > Logs are attached. > > The values we have : > System total memory : 6GB > ES\_HEAP\_SIZE=3g > > We are almost sure this issue comes from long GC run. > We are planning to change the GC for G1 (after upgrading to Java 7u25, > because this GC requires Java 7u4), but I've seen in this group one thread > saying it crashes:/ > What can we do to prevent this behavior and run ES smoothly ? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 14, 2013, 10:04pm UTC](https://discuss.elastic.co/t/garbage-collector-issues/14407/3 "2013-11-14T22:04:42Z")
You probably also want to change to Oracle java as well, OpenJDK is not recommended. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 15 November 2013 04:42, Radu Gheorghe [radu.gheorghe@sematext.com](mailto:radu.gheorghe@sematext.com) wrote: > Hello Vincent, > > If the search threadpool hits the limit maybe you have too many concurrent > searches. If that's the case, you'll probably have to just add nodes and/or > increase the number of replicas. Or, you can look at making your queries > faster, if that's possible. > > G1 may help, I would try it and see how it goes. > > Last but not least, I would look at what is consuming memory. Is it field > cache? Is it filter cache? I think nodes stats can tell you that, and you > could turn a few knobs there and limit memory usage: > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html) > > You may also want to try out our SPM for Elasticsearch. It will show you > all sorts of metrics, from Garbage Collector and pool sizes to cache sizes. > I assume it would be very helpful in this particular case: > [Elasticsearch Monitoring](http://sematext.com/spm/elasticsearch-performance-monitoring/) > > ## Best regards, Radu > > Performance Monitoring \* Log Analytics \* Search Analytics > Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) > > On Thu, Nov 14, 2013 at 6:06 AM, vincent miszczak \< > [vincent.miszczak@gmail.com](mailto:vincent.miszczak@gmail.com)\> wrote: > > > Hi, > > > > We are running Elasticsearch 0.90.2 on Debian 7.0/OpenJDK7u3 (2 nodes > > cluster). > > From time to time, Elasticsearch stop responding and the issue looks > > related to the Garbage Collector. > > > > Here are the information we have collected when problems occur : > > -The search threadpool hits the concurrent active items limit and the > > queue limit (default values, ie 36 threads and 1000 slots in the queue). > > -We have high rate of slow queries (\>8 seconds) > > -The garbage collector logs long passes (around 6 seconds) > > -Clients get Rejected exceptions > > All of this happens for several minutes (\> 10) from time to time > > > > Then everything get back to normal. > > Logs are attached. > > > > The values we have : > > System total memory : 6GB > > ES\_HEAP\_SIZE=3g > > > > We are almost sure this issue comes from long GC run. > > We are planning to change the GC for G1 (after upgrading to Java 7u25, > > because this GC requires Java 7u4), but I've seen in this group one thread > > saying it crashes:/ > > What can we do to prevent this behavior and run ES smoothly ? > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [November 15, 2013, 8:20am UTC](https://discuss.elastic.co/t/garbage-collector-issues/14407/4 "2013-11-15T08:20:14Z")
If you update to ES 0.90.7 or later, you should be safe in using the G1 GC collector, because the GNU Trove collections have been replaced by HPPC. I agree, moving from OpenJDK to the latest Oracle Java (7u25 is known to be stable) can help in erratic memory situations. But first, you have to check what is reason why you allocate so much data on the heap so that it can not be garbage collected. Maybe simply your requirements are too high for just 2 nodes and you need to add more nodes. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![vincent\_miszczak](https://avatars.discourse-cdn.com/v4/letter/v/b77776/32.png) [@vincent\_miszczak](https://discuss.elastic.co/u/vincent_miszczak)\ **Post date:** [November 15, 2013, 1:07pm UTC](https://discuss.elastic.co/t/garbage-collector-issues/14407/5 "2013-11-15T13:07:06Z")
Hi guys, Thank you for your help. From your advice, we're gonna test ES 0.90.7+, Oracle JDK, G1 and look what is consuming memory. This will take a few days to setup the environment, I'll come back with results when I have them. About the load, our CPU are most of the time idle. Would having more memory help to have a better behavior ? About Oracle/OpenJDK, is there a real difference between the 2 products behaviour with ES ? OpenJDK ships builtin into Debian, while OracleJDK don't. We like the idea to simply apt-get upgrade the package to get the latest patches (and Java is very often patched). Vincent We're Le jeudi 14 novembre 2013 15:06:56 UTC+1, vincent miszczak a écrit : > Hi, > > We are running Elasticsearch 0.90.2 on Debian 7.0/OpenJDK7u3 (2 nodes > cluster). > From time to time, Elasticsearch stop responding and the issue looks > related to the Garbage Collector. > > Here are the information we have collected when problems occur : > -The search threadpool hits the concurrent active items limit and the > queue limit (default values, ie 36 threads and 1000 slots in the queue). > -We have high rate of slow queries (\>8 seconds) > -The garbage collector logs long passes (around 6 seconds) > -Clients get Rejected exceptions > All of this happens for several minutes (\> 10) from time to time > > Then everything get back to normal. > Logs are attached. > > The values we have : > System total memory : 6GB > ES\_HEAP\_SIZE=3g > > We are almost sure this issue comes from long GC run. > We are planning to change the GC for G1 (after upgrading to Java 7u25, > because this GC requires Java 7u4), but I've seen in this group one thread > saying it crashes:/ > What can we do to prevent this behavior and run ES smoothly ? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [November 15, 2013, 2:21pm UTC](https://discuss.elastic.co/t/garbage-collector-issues/14407/6 "2013-11-15T14:21:05Z")
It depends on the OpenJDK release version. You have mentioned OpenJDK 7u3, and this is an older version, maybe with open bugs affecting Lucene/ES. OpenJDK forms the base for Oracle JDK [http://openjdk.java.net/projects/jdk7u/qanda.html](http://openjdk.java.net/projects/jdk7u/qanda.html) Vendors and distributors may patch OpenJDK for their purposes, and they also recommend Java versions. It is up to you to get informed about the best solution for you. I always recommend to update to the latest Java 7 version, because of the chance to get most known bugs fixed. Please note that every once a while, new Java releases bring new challenges to run Lucene/ES smoothly. There is unfortunately no "official" certification process of finding a reliable JVM for Lucene/ES to mitigate risks, only advise from best practice is available. In general, all JVMs since version 6 should be able to run ES "somehow" (ie. without crashing). Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![vincent\_miszczak](https://avatars.discourse-cdn.com/v4/letter/v/b77776/32.png) [@vincent\_miszczak](https://discuss.elastic.co/u/vincent_miszczak)\ **Post date:** [December 9, 2013, 1:39pm UTC](https://discuss.elastic.co/t/garbage-collector-issues/14407/7 "2013-12-09T13:39:22Z")
Hi, Some feedback for the community. We have upgraded to ES 0.90.7+, same GC problems. We have upgraded OpenJDK 7 from update 3 to update 25 : 1. we have a 2 nodes cluster, and running u3 aside with u25 give serialization errors, we needed to upgrade both hosts at the same time. 2. we got strange results with u25. We had large CPU usage and ES regularly stopped responding because of this. We have upgraded to Oracle JDK 7 update 45. No more CPU issue, no problem for now, no more GC issues without more tuning. We are still watching if GC behaves correctly but behaviour looks much better. Vincent -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e255ab6b-385f-47ee-a8f6-c5b95f12069d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e255ab6b-385f-47ee-a8f6-c5b95f12069d%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:02am UTC](https://discuss.elastic.co/t/garbage-collector-issues/14407/8 "2017-07-06T02:02:28Z")