# Garbage collector taking more time with increase in search requests **URL:** \ **Category:** Elasticsearch\ **Created:** [April 19, 2014, 8:15am UTC](https://discuss.elastic.co/t/garbage-collector-taking-more-time-with-increase-in-search-requests/17095 "2014-04-19T08:15:06Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Rajkumar\_S](https://avatars.discourse-cdn.com/v4/letter/r/fbc32d/32.png) [@Rajkumar\_S](https://discuss.elastic.co/u/Rajkumar_S)\ **Post date:** [April 19, 2014, 8:15am UTC](https://discuss.elastic.co/t/garbage-collector-taking-more-time-with-increase-in-search-requests/17095/1 "2014-04-19T08:15:06Z")
Hi, We have an elasticsearch cluster running in production with the below configuration. Our cluster has a master-data node topology with 8 master nodes serving traffic. The issue we are facing very recently is that garbage collector is running more frequently and also taking more time as well(than usual) for the past few days. Total garbage collection on all master nodes takes a maximum of around 250s to complete. Search requests have increased a bit of late and we are still using nio fs as the index store type([http://www.elasticsearch.org/guide/en/elasticsearch/reference/0.90/index-modules-store.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/0.90/index-modules-store.html) ). This is affecting our latencies and ultimately requests get dropped once the cluster is not able to handle the traffic. We have a queue on top of incoming requests and is constantly at ~10. Most of our requests are either get/bulkUpdate/search. Configuration: - "number\_of\_nodes": 17, - "number\_of\_data\_nodes": 9, - "active\_primary\_shards": 24730, - "active\_shards": 74196, - Elasticsearch version: 0.90.11 - Java: 1.7u55 - Garbage collector: G1 We have one index per user and one shard+one replica for each index. Current number of indices is 24730 and we expect it to go as much as 45000. We created around 10k indices in the last month. And we just upgraded to java 1.7u55 from 1.7u51. Our master nodes have 64gb of RAM and elasticsearch is using half of it. So how does search requests affect heap size and garbage collection? We do not have any cross index search requests. What can be the possible reasons for GC taking so much time in the past few days (even before the java upgrade as well)? Can we use mmapfs so that less heap size gets used and hence GC could run faster? Thanks, Raj -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/438e067c-a50b-445e-ae3a-ac1c3a04bec4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/438e067c-a50b-445e-ae3a-ac1c3a04bec4%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 19, 2014, 9:34am UTC](https://discuss.elastic.co/t/garbage-collector-taking-more-time-with-increase-in-search-requests/17095/2 "2014-04-19T09:34:56Z")
mmapfs is great on x64, and if you can upgrade to 1.X. But you may just be reaching the limits of the cluster, therefore need more nodes. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 19 April 2014 18:15, Rajkumar S [rajkumarsceg@gmail.com](mailto:rajkumarsceg@gmail.com) wrote: > Hi, > > We have an elasticsearch cluster running in production with the below > configuration. Our cluster has a master-data node topology with 8 master > nodes serving traffic. The issue we are facing very recently is that > garbage collector is running more frequently and also taking more time as > well(than usual) for the past few days. Total garbage collection on all > master nodes takes a maximum of around 250s to complete. Search requests > have increased a bit of late and we are still using nio fs as the index > store type( > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/0.90/index-modules-store.html)). This is affecting our latencies and ultimately requests get dropped once > the cluster is not able to handle the traffic. We have a queue on top of > incoming requests and is constantly at ~10. Most of our requests are either > get/bulkUpdate/search. > > Configuration: > > - "number\_of\_nodes": 17, > - "number\_of\_data\_nodes": 9, > - "active\_primary\_shards": 24730, > - "active\_shards": 74196, > - Elasticsearch version: 0.90.11 > - Java: 1.7u55 > - Garbage collector: G1 > > We have one index per user and one shard+one replica for each index. > Current number of indices is 24730 and we expect it to go as much as 45000. > We created around 10k indices in the last month. And we just upgraded to > java 1.7u55 from 1.7u51. > > Our master nodes have 64gb of RAM and elasticsearch is using half of it. > > So how does search requests affect heap size and garbage collection? We do > not have any cross index search requests. What can be the possible reasons > for GC taking so much time in the past few days (even before the java > upgrade as well)? Can we use mmapfs so that less heap size gets used and > hence GC could run faster? > > Thanks, > > Raj > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/438e067c-a50b-445e-ae3a-ac1c3a04bec4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/438e067c-a50b-445e-ae3a-ac1c3a04bec4%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/438e067c-a50b-445e-ae3a-ac1c3a04bec4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/438e067c-a50b-445e-ae3a-ac1c3a04bec4%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624b3evMsO7BRLv34-jsh%3D%3DZ4\_EiVXt1MJnPHrvFw816Aqw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624b3evMsO7BRLv34-jsh%3D%3DZ4_EiVXt1MJnPHrvFw816Aqw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:34am UTC](https://discuss.elastic.co/t/garbage-collector-taking-more-time-with-increase-in-search-requests/17095/3 "2017-07-06T01:34:51Z")
Closed on July 6, 2017, 1:34am UTC