100% CPU after the upgrade from 7.6.1 to 7.8.1

Jehu_T · August 13, 2020, 8:19am

Hi,

After upgrading to Elasticsearch 7.8.1 from 7.6.1, the CPU usage is reaching 100% most of the time on all 4 nodes. We didn't have this issue before the upgrade.

The GC is almost 0% after the upgrade. It was noticed that the java version has been upgraded to 14. But the jvm.options file is having the config for the previous version. We are yet to apply these changes and would like to know if this GC and high CPU usage are related to this configuration.

Screenshots of Kibana metrics:

Hot Threads:

gist.github.com

https://gist.github.com/devopssupportcapestart/a7c44cabb134c35431844664b37f5235

hot_threads

::: {prod-es01}{cx9cWDAlTWKVjnYrbq8J7w}{gfaIzpyNS9qQpaoi0T-wsw}{192.168.90.11}{192.168.90.11:9300}{dilmrt}{ml.machine_memory=169012310016, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}
   Hot threads at 2020-08-11T10:17:15.112Z, interval=500ms, busiestThreads=99999, ignoreIdleThreads=false:
   
   68.8% (343.8ms out of 500ms) cpu usage by thread 'elasticsearch[prod-es01][search][T#12]'
     3/10 snapshots sharing following 34 elements
       app//org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:823)
       app//org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:823)
       app//org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:506)
       app//org.elasticsearch.search.internal.ExitableDirectoryReader$ExitablePointValues.intersect(ExitableDirectoryReader.java:194)
       app//org.apache.lucene.search.PointRangeQuery$1$4.get(PointRangeQuery.java:311)

This file has been truncated. show original

GC logging is not enabled, so couldn't get it.

JVM config for java 14 (not applied yet):

github.com

elastic/elasticsearch/blob/7.8/distribution/src/config/jvm.options#L46-L48


14-:-XX:+UseG1GC
14-:-XX:G1ReservePercent=25
14-:-XX:InitiatingHeapOccupancyPercent=30

Thanks,
Jehu

Jehu_T · August 25, 2020, 11:02am

I have added the JVM variables for java 14 and still, the CPU utilization is reaching 100% and production application went down as well. We never had this issue with 7.6.1.

We are planning to downgrade to 7.6.1. Would there be any data loss as it's within version 7?
We have 3 master nodes and 1 data node.

Thanks,
Jehu

Christian_Dahlqvist · August 25, 2020, 11:45am

Elasticsearch does not support downgraded at all so you would need to restore data from previous snapshot to do that.

Jehu_T · August 25, 2020, 1:10pm

Ok thanks. Is this performance issue a known issue with version 7.8.1?

Jehu_T · September 16, 2020, 10:42am

I have upgraded the version to 7.9.1 but the issue is still there.

When I calculated the total data nodes required based on the formula mentioned in the document below, the data nodes required is increasing with response time ( Average search response time in milliseconds ). But with more nodes, the response time should reduce. Did I understand this correctly?

Calculation:
Average search response time = 2 sec (2000 milli seconds)
(100x2000)/1000 = 200
(20x2x3/2)+1 = 61
200/31 = 3 nodes

Average search response time = 10 sec (10000 milli seconds)
100x10000/1000 = 1000
(20x2x3/2)+1 = 61
1000/31 = 16 nodes

Also, regarding the heap size, the document says: "This should be 50% of available RAM, and up to a maximum of 30GB RAM to avoid garbage collection." In our case the RAM of the server is 160GB and the heap size is 80GB. Is this a problem?