Investigate high cpu/rogue task on certain shard/node

Elastic Stack Elasticsearch
1

for some reason, some nodes/shards doing more "stuff" than others and thus consuming more cpu.

here is setup

  • ES deployed in kubernetes with 40+ data nodes (with different specs)
  • ES version 8.10.4
  • the nodes are partitioned (using node attributes) into 4 separate node-sets
  • the indexes allocation has been set such that certain node-set only contain certain index
  • there are 18 nodes in set2
  • there is only 1 large index allocated to set2
  • shard allocation is fairly balanced, some have 5 and others 6

the issue

  • 2 of the nodes in set2 experienced elevated cpu usage compared to other nodes in the same set
  • both nodes started experiencing this issue during heavy query/aggregation application load.
  • initially, the cpu usage on all 18 nodes were similar, and then after some time those 2 node started having issues
  • even when the application is idle, those 2 ES nodes are still consuming 2~3 cpu cores. while the rest of nodes in set2 are ~0.5 cores

observation

  • couldnt find anything suspicious in _task or nodes/hot_threads. they seem to do query
  • node restart doesnt reduce the cpu usage. (pod was deleted from kubernetes and let it restart itself)
  • when all shards are moved away (auto rebalance is disabled) from 1 of the problematic node, node-0, the cpu usage of that node goes down and no other node experienced cpu increase.
  • when the underlying kubernetes node, node-1, was replaced , cpu usage goes down to normal level, inline with the rest on the set2 nodes. (the node is cordoned off, new node is provisioned and the pod is deleted)

question

  • what sort of logs/metrics/info needed to investigate this issue?
  • anyone experienced similar issue?

thanks in advance