Help understanding hot threads (CPU 100% for several hours)

Elastic Stack Elasticsearch
1

I indexed a lot of data into my elasticsearch cluster several hours ago. The CPU utilization went to 100% while this was happening, but now that it has finished (about 3 hours ago) the CPU utilization is still 100%. I had the refresh interval est to "-1" during the indexing, but have since set it to "60s".

I have been looking at the output of hot threads, but can't work out what is happening with my cluster, as far as I know there are very few queries going to it now. It seems like a lot of the output is from log4j and netty.

My hot_threads output: https://gist.github.com/carasel/34467b214bd71e9680da3ffa4d8584b2

I have 300 shards (including replication of 1) across 30 indices, 3 master nodes, 4 data nodes.

2
  • node mGhV-nU is busy right now with compressing logs
  • node NeCaeFk is also compressing logs, but also runs a regex query? Just guessing due to your redacted parts, you may want to show those as well.
  • node IjibcFs runs a more like this query, scoring
  • node SSt5weZ is doing some scoring + more like this as well

Are you logging a lot of things? Apart from that we have some normal searches going on, from this 2 minute check.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.