Running 7.6.2 on elastic cloud and observing increasing CPU over time on our cluster.
Some background to our configuration:
2x Nodes x 2 Zone 58GB Data Nodes.
1x Nodes x 2 Zone 8GB ML Nodes
3x Master Nodes at 4GB
2x 4GB coordinator nodes
1x 8GB kibana nodes x 2 zones.
Shards Total: 766
We upgraded to 7.6.2 beginning of May and 15 days was very solid. But now seeing consistently rising CPU
I look at hot threads as I found a similar post here: High CPU Usage on Data Node
I then see stuff like this:
104.3% (521.4ms out of 500ms) cpu usage by thread 'elasticsearch[instance-0000000119] [write][T#13]' 10/10 snapshots sharing following 191 elements app//org.apache.lucene.store.DataInput.readVInt(DataInput.java:126)
Any ideas. I checked ingestion is normal, old indexes are being cleared down regularly etc.