# Production cluster slows down after 15-20 days of starting the services **URL:** \ **Category:** Elasticsearch\ **Created:** [May 3, 2016, 3:04pm UTC](https://discuss.elastic.co/t/production-cluster-slows-down-after-15-20-days-of-starting-the-services/49070 "2016-05-03T15:04:07Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Hanish\_Bansal](https://avatars.discourse-cdn.com/v4/letter/h/e36b37/32.png) [@Hanish\_Bansal](https://discuss.elastic.co/u/Hanish_Bansal)\ **Post date:** [May 3, 2016, 3:04pm UTC](https://discuss.elastic.co/t/production-cluster-slows-down-after-15-20-days-of-starting-the-services/49070/1 "2016-05-03T15:04:08Z")
We are using 5 machines cluster in production for ElasticSearch. We are facing an issue that initially cluster response is good but after 15-20 days of starting the services, ES response get slow down, also indexing process get slow down. To resolve the issue we have to restart ElasticSearch on couple of nodes. Details: ES Version: 1.5.2 RAM per node: 30 GB ES Heap size per node: 12 GB Disk Space per node: 1 TB We are indexing 40 millions records in a day of size 30 GB. Indexes are created on daily basis. Replication factor is 2 as of now. We have to restart the ES after 15-20 days to keep it stable. It would be great help if anyone could share their suggestions ---
**Author:** ![Bruce\_Ritchie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bruce_ritchie/32/9370_2.png) [@Bruce\_Ritchie](https://discuss.elastic.co/u/Bruce_Ritchie)\ **Post date:** [May 3, 2016, 6:43pm UTC](https://discuss.elastic.co/t/production-cluster-slows-down-after-15-20-days-of-starting-the-services/49070/2 "2016-05-03T18:43:01Z")
Do you have gc logging enabled? If so you may want to check to see if ES is slowly eating up it's heap and slowing down because of GC's. There has been a couple of improvements and fixes post 1.5 that may impact your cluster - the one that comes to mind is described here - [JVM heap wasted in Segments fixedBitSet](https://discuss.elastic.co/t/jvm-heap-wasted-in-segments-fixedbitset/48574) ---
**Author:** ![Hanish\_Bansal](https://avatars.discourse-cdn.com/v4/letter/h/e36b37/32.png) [@Hanish\_Bansal](https://discuss.elastic.co/u/Hanish_Bansal)\ **Post date:** [May 5, 2016, 3:22pm UTC](https://discuss.elastic.co/t/production-cluster-slows-down-after-15-20-days-of-starting-the-services/49070/3 "2016-05-05T15:22:30Z")
Hi Bruce, Thanks for reply. I have not enabled gc logging explicitly. Can you please let me know how to enable gc logging? As of now we can not upgrade ES as there are major changes in java apis also. ---
**Author:** ![Bruce\_Ritchie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bruce_ritchie/32/9370_2.png) [@Bruce\_Ritchie](https://discuss.elastic.co/u/Bruce_Ritchie)\ **Post date:** [May 5, 2016, 3:35pm UTC](https://discuss.elastic.co/t/production-cluster-slows-down-after-15-20-days-of-starting-the-services/49070/4 "2016-05-05T15:35:31Z")
API changes while they exist should not be horribly onerous (we did the upgrade from 1.3 to 2.3 in a week or so) especially for a 1.5 -\> 1.7.5. With respect to gc logging I'm pretty sure there is an environment variable to enable it. Check your bin/elasticsearch and/or bin/elasticsearch.in.sh file for details. ---
**Author:** ![Hanish\_Bansal](https://avatars.discourse-cdn.com/v4/letter/h/e36b37/32.png) [@Hanish\_Bansal](https://discuss.elastic.co/u/Hanish_Bansal)\ **Post date:** [May 5, 2016, 5:13pm UTC](https://discuss.elastic.co/t/production-cluster-slows-down-after-15-20-days-of-starting-the-services/49070/5 "2016-05-05T17:13:40Z")
Okay. I will check about GC logging. I am just curious to know, did you not face any issue for data compatibility while upgrading ES from 1.3 to 2.3? I mean data indexed in old version of ES is compatible with new ES version or is there some process for this? ---
**Author:** ![Bruce\_Ritchie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bruce_ritchie/32/9370_2.png) [@Bruce\_Ritchie](https://discuss.elastic.co/u/Bruce_Ritchie)\ **Post date:** [May 5, 2016, 5:15pm UTC](https://discuss.elastic.co/t/production-cluster-slows-down-after-15-20-days-of-starting-the-services/49070/6 "2016-05-05T17:15:41Z")
We have to reindex because we had a field with a period in the name which is disallowed in 2.3. There is a compatibility plugin for 1.x that can tell you if your system can be upgraded in place or not. ---
**Author:** ![Hanish\_Bansal](https://avatars.discourse-cdn.com/v4/letter/h/e36b37/32.png) [@Hanish\_Bansal](https://discuss.elastic.co/u/Hanish_Bansal)\ **Post date:** [May 6, 2016, 3:20pm UTC](https://discuss.elastic.co/t/production-cluster-slows-down-after-15-20-days-of-starting-the-services/49070/7 "2016-05-06T15:20:41Z")
Thanks Bruce for this information. Is there any REST api in ES, using that we can identify heap usage like which is eating up the memory? ---
**Author:** ![Bruce\_Ritchie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bruce_ritchie/32/9370_2.png) [@Bruce\_Ritchie](https://discuss.elastic.co/u/Bruce_Ritchie)\ **Post date:** [May 6, 2016, 3:54pm UTC](https://discuss.elastic.co/t/production-cluster-slows-down-after-15-20-days-of-starting-the-services/49070/8 "2016-05-06T15:54:00Z")
I'd suggest looking at the stats api - [https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html) [https://www.elastic.co/guide/en/elasticsearch/guide/current/\_monitoring\_individual\_nodes.html](https://www.elastic.co/guide/en/elasticsearch/guide/current/_monitoring_individual_nodes.html) If that doesn't help than perhaps a heap dump + eclipse memory analyzer. Be aware though that you'll need a decent understanding of ES's internals to make sense of the classes and their hierarchy. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:53pm UTC](https://discuss.elastic.co/t/production-cluster-slows-down-after-15-20-days-of-starting-the-services/49070/9 "2017-07-05T22:53:17Z")