What's with these crazy CPU spikes?

[sysadmin]

Hey, I'm relatively new to ES and am building out a cluster for some pretty heavy lifting. Throughout my whole process of building and troubleshooting this thing so far, there's been a pattern of spikes to almost 100% CPU load every 5 minutes on all of my nodes. The graph only shows it happening on some each time, but that's just the 1 minute check interval on my metrics agent not catching the short event on each node each time.

Can anyone help me identify what's causing this? Is it normal or something I should be trying to fix? I can tell you it's the Java process on the node doing it, not some other process.

What troubleshooting steps can I take? What info can I provide here? I did some struggling with X-Pack today but was not able to get it running (and trying is pretty disruptive). Is that the right tool?

I've been stuck at this for a couple days. Can anyone provide any input?

Thanks

[jakelandis]

What troubleshooting steps can I take?

Nodes hot threads API | Elasticsearch Guide [8.11] | Elastic while the spike is happening is probably the best start. The results may be pretty hard to understand since they are Java stack traces, however the class names in them can often point you a direction to start investigating.