ElasticSearch 7.4 taking 100% cpu usage

Elastic Stack Elasticsearch
1

Recently we migrated from ES-5.1 to ES7.4 and the CPU usage too high in ES7.4 compared to ES5.1

With a given load the max. cpu usage of 5.1 is in between 0% to 10%.
However the performance and max. cpu usage of 7.4 is in between 20% to 100%.
The cluster configuration of 7.4 is exactly same as 5.1 and same queries being used.

Please find the hot threads output for ES7.4
When there is no load
https://gist.github.com/pavanladi98/ad867151bc2f5b2803f724dc8d3493db
When there is usual load
https://gist.github.com/pavanladi98/dc5cab94be43dc2771ef8362bccc4a7a

2

hey,

can you go into some more details about your query load? What kind fo query are you running? Does it include an aggregation? Do you have any special configuration for global ordinals in your mapping? Do you use any special fields? Is this a heavily updated index?

Some more context would be great.

--Alex

3

Hi, thanks for the reply

  1. We mostly use a function_score query with some filters on geolocation, date etc... fileds.
  2. We also use another boolean match query along with aggregations.

The data and queries we used are same in 5.1 and 7.4.

But in 7.4 ,
To improve the search speed of this aggregation query, we used Eager_global_ordinals: True on few fields. Since this might be costly at indexing time, as suggested in elasticsearch documentaion we are using refresh_interval: 30s .

Yes, our data gets indexed/updated frequently but at slow rate, approximately at rate of 20 docs per minute.

Other differences:

  1. we enabled encryption(REST encryption and node-to-node encryption) on our 7.4 elasticsearch cluster.
  2. we increased number of shards for our largest index(pri.size = 600GB) in 7.4
    number_of_shards: 6 (in 5.1)
    number_of_shards: 15 (in 7.4)
4

The important part about global ordinals is in the last paragraph

Because global ordinals provide a unified mapping for all segments on the shard, they also need to be rebuilt entirely when a new segment becomes visible.

You may want to try a different execution_hint and see what happens.

Just to be sure, did you use eager loading of ordinals in 5.1 as well or added it in 7.4?

5

We used Eager_global_ordinals: true in mapping only in 7.4

We had millions of docs matching the query. So, as suggested we are using default excecution_hint

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.