High CPU on Data Nodes

[Wayne_Taylor]

Hi All,

Running 7.6.2 on elastic cloud and observing increasing CPU over time on our cluster.

Some background to our configuration:
2x Nodes x 2 Zone 58GB Data Nodes.
1x Nodes x 2 Zone 8GB ML Nodes
3x Master Nodes at 4GB
2x 4GB coordinator nodes
1x 8GB kibana nodes x 2 zones.

Indices: 222
Shards Total: 766

We upgraded to 7.6.2 beginning of May and 15 days was very solid. But now seeing consistently rising CPU

I look at hot threads as I found a similar post here: High CPU Usage on Data Node

I then see stuff like this:
104.3% (521.4ms out of 500ms) cpu usage by thread 'elasticsearch[instance-0000000119] [write][T#13]' 10/10 snapshots sharing following 191 elements app//org.apache.lucene.store.DataInput.readVInt(DataInput.java:126)

Any ideas. I checked ingestion is normal, old indexes are being cleared down regularly etc.

[warkolm (Mark Walkom)]

What does your GC look like?

[Wayne_Taylor]

Here you go @warkolm

It seems to be trending ok.