Hey I'm currently experiencing a high CPU usage in environment. This is my result of hot threads any idea why the spike is happening ?
You have a lot of org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:47) that comes from the indices stats API. I wouldn't expect more than load average of 1 given what I see on the there. Depending on the version this could be caused by a bug we've since fixed. If you are seeing more than a load average of around 1 then it is something that isn't on the hot threads.
By "caused by a bug we've since fixed" I mean that there were versions of Elasticsearch in the 1.x line that were fairly slow at running that request. But it should only come up if you are frequently making that request. Is your monitoring system doing it?
Hey nik9000 thank you for your response. I just checked my version and I'm running the latest 2.3.4 I also just checked to see if I had a tab open with my monitor plugin but I did not .
I run kopf and hq as my monitoring plugins
I pulled another hot thread dump using kopf
number of threads: 10
I can paste my node stats if you need to see more information
That last one just has a single thread doing things. It is busy, but I think it is cut off so I can't tell exactly what. Looks statistics related. What kind of load average are you seeing? How many shards and things?
2.3.4 also has the tasks API so you can look there at running tasks and see if you see any stats requests.
So I think I found the issue. my ES node were running out of heap I allocated another Gig to all the data nodes and the CPU usage dropped by 50%.
Thank you for your help!