# 100% CPU after the upgrade from 7.6.1 to 7.8.1

**URL:** <https://discuss.elastic.co/t/100-cpu-after-the-upgrade-from-7-6-1-to-7-8-1/244822>\
**Category:** Elasticsearch\
**Created:** [August 13, 2020, 8:14am UTC](https://discuss.elastic.co/t/100-cpu-after-the-upgrade-from-7-6-1-to-7-8-1/244822 "2020-08-13T08:14:32Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jehu\_T](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jehu_t/32/64590_2.png) [@Jehu\_T](https://discuss.elastic.co/u/Jehu_T)\
**Post date:** [August 13, 2020, 8:14am UTC](https://discuss.elastic.co/t/100-cpu-after-the-upgrade-from-7-6-1-to-7-8-1/244822/1 "2020-08-13T08:14:33Z")

</div>

Hi,

After upgrading to Elasticsearch 7.8.1 from 7.6.1, the CPU usage is reaching 100% most of the time on all 4 nodes. We didn't have this issue before the upgrade.

The GC is almost 0% after the upgrade. It was noticed that the java version has been upgraded to 14. But the jvm.options file is having the config for the previous version. We are yet to apply these changes and would like to know if this GC and high CPU usage are related to this configuration.

Screenshots of Kibana metrics:

 ![Metrics1](https://us1.discourse-cdn.com/elastic/original/3X/a/9/a9208b3947ec497b836f96cbc02700748e8d1e13.png) ![Metrics2](https://us1.discourse-cdn.com/elastic/original/3X/b/a/baa0bc72b38a7504fde027e9a9a6c7a931fcc665.png)

Hot Threads:

> <https://gist.github.com/devopssupportcapestart/a7c44cabb134c35431844664b37f5235>

GC logging is not enabled, so couldn't get it.

JVM config for java 14 (not applied yet):

> <https://github.com/elastic/elasticsearch/blob/7.8/distribution/src/config/jvm.options#L46-L48>

Thanks,  
Jehu

---

<div class="post-metadata">

**Author:** ![Jehu\_T](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jehu_t/32/64590_2.png) [@Jehu\_T](https://discuss.elastic.co/u/Jehu_T)\
**Post date:** [August 25, 2020, 10:45am UTC](https://discuss.elastic.co/t/100-cpu-after-the-upgrade-from-7-6-1-to-7-8-1/244822/2 "2020-08-25T10:45:21Z")

</div>

I have added the JVM variables for java 14 and still, the CPU utilization is reaching 100% and production application went down as well. We never had this issue with 7.6.1.

We are planning to downgrade to 7.6.1. Would there be any data loss as it's within version 7?  
We have 3 master nodes and 1 data node.

Thanks,  
Jehu

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 25, 2020, 11:45am UTC](https://discuss.elastic.co/t/100-cpu-after-the-upgrade-from-7-6-1-to-7-8-1/244822/3 "2020-08-25T11:45:45Z")

</div>

Elasticsearch does not support downgraded at all so you would need to restore data from previous snapshot to do that.

---

<div class="post-metadata">

**Author:** ![Jehu\_T](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jehu_t/32/64590_2.png) [@Jehu\_T](https://discuss.elastic.co/u/Jehu_T)\
**Post date:** [August 25, 2020, 1:10pm UTC](https://discuss.elastic.co/t/100-cpu-after-the-upgrade-from-7-6-1-to-7-8-1/244822/4 "2020-08-25T13:10:10Z")

</div>

Ok thanks. Is this performance issue a known issue with version 7.8.1?

---

<div class="post-metadata">

**Author:** ![Jehu\_T](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jehu_t/32/64590_2.png) [@Jehu\_T](https://discuss.elastic.co/u/Jehu_T)\
**Post date:** [September 16, 2020, 10:39am UTC](https://discuss.elastic.co/t/100-cpu-after-the-upgrade-from-7-6-1-to-7-8-1/244822/5 "2020-09-16T10:39:48Z")

</div>

I have upgraded the version to 7.9.1 but the issue is still there.

When I calculated the total data nodes required based on the formula mentioned in the document below, the data nodes required is increasing with response time ( Average search response time in milliseconds ). But with more nodes, the response time should reduce. Did I understand this correctly?

> **[elasticsearch-sizing-and-capacity-planning.pdf](https://www.elastic.co/pdf/elasticsearch-sizing-and-capacity-planning.pdf)**
>
> 622.17 KB

  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/2/72d516d0664f3d4fc6bbe73977a3bff31caf6403.png)

Calculation:  
Average search response time = 2 sec (2000 milli seconds)  
(100x2000)/1000 = 200  
(20x2x3/2)+1 = 61  
200/31 = 3 nodes

Average search response time = 10 sec (10000 milli seconds)  
100x10000/1000 = 1000  
(20x2x3/2)+1 = 61  
1000/31 = 16 nodes

Also, regarding the heap size, the document says: "This should be 50% of available RAM, and up to a maximum of 30GB RAM to avoid garbage collection." In our case the RAM of the server is 160GB and the heap size is 80GB. Is this a problem?

---

<div class="post-metadata">

**Author:** ![JohannesMahne](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johannesmahne/32/53911_2.png) [@JohannesMahne](https://discuss.elastic.co/u/JohannesMahne)\
**Post date:** [September 30, 2020, 10:29am UTC](https://discuss.elastic.co/t/100-cpu-after-the-upgrade-from-7-6-1-to-7-8-1/244822/6 "2020-09-30T10:29:57Z")

</div>

Please review [our documentation on setting the correct heap space value](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/heap-size.html) again. Note this:

> Set `Xmx` and `Xms` to no more than the threshold that the JVM uses for compressed object pointers (compressed oops); the exact threshold varies but is near 32 GB.

If you use 80gb of heap space, you will not be using compressed object pointers and will face issues.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 28, 2020, 10:29am UTC](https://discuss.elastic.co/t/100-cpu-after-the-upgrade-from-7-6-1-to-7-8-1/244822/7 "2020-10-28T10:29:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
