# ES version : 5.1.2 - Retained heap is keep on increasing - causing more GCs after some time - increasing CPU spike

**URL:** <https://discuss.elastic.co/t/es-version-5-1-2-retained-heap-is-keep-on-increasing-causing-more-gcs-after-some-time-increasing-cpu-spike/135686>\
**Category:** Elasticsearch\
**Created:** [June 13, 2018, 10:36am UTC](https://discuss.elastic.co/t/es-version-5-1-2-retained-heap-is-keep-on-increasing-causing-more-gcs-after-some-time-increasing-cpu-spike/135686 "2018-06-13T10:36:21Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![srikhark](https://avatars.discourse-cdn.com/v4/letter/s/97f17d/32.png) [@srikhark](https://discuss.elastic.co/u/srikhark)\
**Post date:** [June 13, 2018, 10:36am UTC](https://discuss.elastic.co/t/es-version-5-1-2-retained-heap-is-keep-on-increasing-causing-more-gcs-after-some-time-increasing-cpu-spike/135686/1 "2018-06-13T10:36:21Z")

</div>

> <https://github.com/elastic/elasticsearch/issues/31292>

Can you go through this and give any updates for this?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 13, 2018, 10:38am UTC](https://discuss.elastic.co/t/es-version-5-1-2-retained-heap-is-keep-on-increasing-causing-more-gcs-after-some-time-increasing-cpu-spike/135686/2 "2018-06-13T10:38:14Z")

</div>

Can you please provide the full output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/cluster-stats.html)?

---

<div class="post-metadata">

**Author:** ![srikhark](https://avatars.discourse-cdn.com/v4/letter/s/97f17d/32.png) [@srikhark](https://discuss.elastic.co/u/srikhark)\
**Post date:** [June 13, 2018, 1:04pm UTC](https://discuss.elastic.co/t/es-version-5-1-2-retained-heap-is-keep-on-increasing-causing-more-gcs-after-some-time-increasing-cpu-spike/135686/3 "2018-06-13T13:04:33Z")

</div>

{  
"\_nodes": {  
"total": 3,  
"successful": 3,  
"failed": 0  
},  
"cluster\_name": "XXXXXXXX",  
"timestamp": 1528894949232,  
"status": "green",  
"indices": {  
"count": 26,  
"shards": {  
"total": 178,  
"primaries": 86,  
"replication": 1.069767441860465,  
"index": {  
"shards": {  
"min": 6,  
"max": 10,  
"avg": 6.846153846153846  
},  
"primaries": {  
"min": 3,  
"max": 5,  
"avg": 3.3076923076923075  
},  
"replication": {  
"min": 1,  
"max": 2,  
"avg": 1.0769230769230769  
}  
}  
},  
"docs": {  
"count": 3565594,  
"deleted": 213371  
},  
"store": {  
"size": "104.5gb",  
"size\_in\_bytes": 112247772532,  
"throttle\_time": "0s",  
"throttle\_time\_in\_millis": 0  
},  
"fielddata": {  
"memory\_size": "1.1mb",  
"memory\_size\_in\_bytes": 1199920,  
"evictions": 0  
},  
"query\_cache": {  
"memory\_size": "71.3mb",  
"memory\_size\_in\_bytes": 74815152,  
"total\_count": 280366128,  
"hit\_count": 89906010,  
"miss\_count": 190460118,  
"cache\_size": 52293,  
"cache\_count": 315282,  
"evictions": 262989  
},  
"completion": {  
"size": "0b",  
"size\_in\_bytes": 0  
},  
"segments": {  
"count": 1238,  
"memory": "173.6mb",  
"memory\_in\_bytes": 182122800,  
"terms\_memory": "134.8mb",  
"terms\_memory\_in\_bytes": 141404856,  
"stored\_fields\_memory": "7.7mb",  
"stored\_fields\_memory\_in\_bytes": 8122984,  
"term\_vectors\_memory": "466.5kb",  
"term\_vectors\_memory\_in\_bytes": 477704,  
"norms\_memory": "10.3mb",  
"norms\_memory\_in\_bytes": 10823936,  
"points\_memory": "1.2mb",  
"points\_memory\_in\_bytes": 1261336,  
"doc\_values\_memory": "19.1mb",  
"doc\_values\_memory\_in\_bytes": 20031984,  
"index\_writer\_memory": "0b",  
"index\_writer\_memory\_in\_bytes": 0,  
"version\_map\_memory": "0b",  
"version\_map\_memory\_in\_bytes": 0,  
"fixed\_bit\_set": "236.7kb",  
"fixed\_bit\_set\_memory\_in\_bytes": 242480,  
"max\_unsafe\_auto\_id\_timestamp": -1,  
"file\_sizes": {}  
}  
},  
"nodes": {  
"count": {  
"total": 3,  
"data": 3,  
"coordinating\_only": 0,  
"master": 3,  
"ingest": 3  
},  
"versions": [  
"5.1.2"  
],  
"os": {  
"available\_processors": 48,  
"allocated\_processors": 48,  
"names": [  
{  
"name": "Linux",  
"count": 3  
}  
],  
"mem": {  
"total": "90.2gb",  
"total\_in\_bytes": 96943034368,  
"free": "1.4gb",  
"free\_in\_bytes": 1567109120,  
"used": "88.8gb",  
"used\_in\_bytes": 95375925248,  
"free\_percent": 2,  
"used\_percent": 98  
}  
},  
"process": {  
"cpu": {  
"percent": 16  
},  
"open\_file\_descriptors": {  
"min": 955,  
"max": 962,  
"avg": 958  
}  
},  
"jvm": {  
"max\_uptime": "21.1h",  
"max\_uptime\_in\_millis": 76133306,  
"versions": [  
{  
"version": "1.8.0\_45",  
"vm\_name": "Java HotSpot(TM) 64-Bit Server VM",  
"vm\_version": "25.45-b02",  
"vm\_vendor": "Oracle Corporation",  
"count": 3  
}  
],  
"mem": {  
"heap\_used": "18gb",  
"heap\_used\_in\_bytes": 19396256344,  
"heap\_max": "47.6gb",  
"heap\_max\_in\_bytes": 51199475712  
},  
"threads": 561  
},  
"fs": {  
"total": "885.4gb",  
"total\_in\_bytes": 950796226560,  
"free": "625.9gb",  
"free\_in\_bytes": 672104439808,  
"available": "625.6gb",  
"available\_in\_bytes": 671796477952  
},  
"plugins": [  
{  
"name": "analysis-icu",  
"version": "5.1.2",  
"description": "The ICU Analysis plugin integrates Lucene ICU module into elasticsearch, adding ICU relates analysis components.",  
"classname": "org.elasticsearch.plugin.analysis.icu.AnalysisICUPlugin"  
}  
],  
"network\_types": {  
"transport\_types": {  
"netty4": 3  
},  
"http\_types": {  
"netty4": 3  
}  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 13, 2018, 1:17pm UTC](https://discuss.elastic.co/t/es-version-5-1-2-retained-heap-is-keep-on-increasing-causing-more-gcs-after-some-time-increasing-cpu-spike/135686/4 "2018-06-13T13:17:20Z")

</div>

That looks fine. Can you zoom in on the GC graph you provided and show us here? It would be good if we could see the pattern and GC frequency across a shorter time period, e.g. an hour or two.

---

<div class="post-metadata">

**Author:** ![srikhark](https://avatars.discourse-cdn.com/v4/letter/s/97f17d/32.png) [@srikhark](https://discuss.elastic.co/u/srikhark)\
**Post date:** [June 13, 2018, 1:24pm UTC](https://discuss.elastic.co/t/es-version-5-1-2-retained-heap-is-keep-on-increasing-causing-more-gcs-after-some-time-increasing-cpu-spike/135686/5 "2018-06-13T13:24:04Z")

</div>

![4](https://us1.discourse-cdn.com/elastic/original/3X/1/8/18c47cd466dfcc16cef20782d279a5257e087fa4.png)

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 13, 2018, 1:50pm UTC](https://discuss.elastic.co/t/es-version-5-1-2-retained-heap-is-keep-on-increasing-causing-more-gcs-after-some-time-increasing-cpu-spike/135686/6 "2018-06-13T13:50:28Z")

</div>

Doesn't look too bad, nor exceptionally frequent, but I can see the base is creeping upwards. There has been some [memory leak issues](https://www.elastic.co/blog/tracking-down-native-memory-leaks-in-elasticsearch) fixed since the release you are running, so I would recommend upgrading to version 5.6.9.

---

<div class="post-metadata">

**Author:** ![srikhark](https://avatars.discourse-cdn.com/v4/letter/s/97f17d/32.png) [@srikhark](https://discuss.elastic.co/u/srikhark)\
**Post date:** [June 13, 2018, 1:50pm UTC](https://discuss.elastic.co/t/es-version-5-1-2-retained-heap-is-keep-on-increasing-causing-more-gcs-after-some-time-increasing-cpu-spike/135686/7 "2018-06-13T13:50:58Z")

</div>

I can see 2.2 GB unrachable objects .. after that. i could see 2.7GB unreachable objects in heap dumps

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 13, 2018, 2:27pm UTC](https://discuss.elastic.co/t/es-version-5-1-2-retained-heap-is-keep-on-increasing-causing-more-gcs-after-some-time-increasing-cpu-spike/135686/8 "2018-06-13T14:27:35Z")

</div>

Sounds like you may be affected by one of the memory leaks that has since been fixed, which is why I am recommending upgrading.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 11, 2018, 2:27pm UTC](https://discuss.elastic.co/t/es-version-5-1-2-retained-heap-is-keep-on-increasing-causing-more-gcs-after-some-time-increasing-cpu-spike/135686/9 "2018-07-11T14:27:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.