Elasticsearch 7.12 High CPU usage

Elastic Stack Elasticsearch
1

Hi all,
Last week in need to install a new cluster and I decided to use the 7.12 version.
Initially I've configure a three node cluster (two data+master nodes, one master node).
In this version elasticsearch automatically sizes JVM heap based on a node’s and total memory. So I decided to leave the setting like that.
Surprisingly without executing any query nor indexing data (or indexing very few documents) the nodes ran out of heap memory.
I open an slack thread and I decided to setup the value in the jvm.options as I have in my other ES instances (since apparently in some circumstances may be a problem with that)

This problem was solved but an odd one appeared... the CPU usage was very very high, no matter if I was doing nothing with that node.
I have to downgrade to 7.11.2, using the same configuration an the performance improvement was huge

I decide then to install again (in another server) version 7.12 and perform a side by side comparison.
Below the results

Have any performance problems been reported with this version?

Thank you very much
Regards
Ana

Server 1 (Good Performance)
CPU: 16
RAM: 16GB
Elasticsearch version: 7.11.2
Heap Size: 8GB
Average CPU Utilization: 2%
Average Heap Utilization: 4.7GB
System Load: 1m 1.11

Server 2 (Very Poor Performace)
CPU: 20
RAM: 24GB
Elasticsearch version: 7.12.0
Heap Size: 8GB
Average CPU Utilization: 47%
Average Heap Utilization: 4.1GB
System Load: 1m 8.61

Both server are running ES+Kibana+LS and ingesting the same data.
(Servers are for comparison and testing purpose, that is why I put ES and LS in the same servers)
The Kibana space in use holds the same dashboards and visualizations
Attached the output of the thread_pool, hot_threads, tasks (I put the response time in order to show you the difference in the response time)

Task, Thread Pool, and Hot Threads

1 Like
2

You haven't provided logs, which would be useful here.

(Please also use gist/pastebin/etc instead of gdrive :slight_smile:)

3

Hi @warkolm ,
Logs from today. The curious fact is thet when I stopped logstash the CPU rocketed up :face_with_raised_eyebrow:

image
image942×245 38.2 KB

Now I think there is more information information in the Tasks, Hot Threads, Thread Pool output

Also the here the elasticsearch.log

Any feedback will be welcome
Thank you!
Regards
Ana

4

I can see a spike in GC in your logs, but it's a once off.

What is the output from the _cluster/stats?pretty&human API?

5

Hi,
Here the output _cluster/stats?pretty&human
The response time for returning this information was image

Whereas with the node in 7.11.2 the response time is image

Regards
Ana

6

Hi @warkolm
Today I've updated to 7.12.1 and now it seems that the performance problem is solved
The cluster stats now responds in image
And the CPU and memory usage seems normal now (and consistent with the data and resources I have).
CPU before and after update
image

I read the release notes of version 7.12.1 but I was unable to find anything related to performance or cpu utilization improvements

Regards
Ana

1 Like
7

Ok that's weird but good, thanks for sharing your solution.

8

Yes, the weirdest thing is that It was not my only instance with that problem.
I had a 3 node cluster that was initially installed with 7.12.0 and after see how badly was performing, I decided then to downgrade...
Thanks @warkolm for your support

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.