CPU running high only on one node at time in the ES cluster

Elastic Stack Elasticsearch
1

Not sure why only one node in the cluster has such high CPU when there is nothing happening in the cluster.

top - 00:34:43 up 137 days, 17:43,  1 user,  load average: 1.24, 2.14, 3.15
Tasks: 198 total,   1 running, 197 sleeping,   0 stopped,   0 zombie
%Cpu(s): 27.7 us,  1.9 sy,  0.0 ni, 68.6 id,  1.7 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem : 61676952 total,  1712800 free, 34528200 used, 25435952 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 26000248 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 8302 elastic+  20   0  297.6g  38.9g   6.9g S **476.1** 66.1 252422:37 java
  567 root      20   0       0      0      0 S   1.3  0.0 971:45.39 jbd2/dm-0-8
  106 root      20   0       0      0      0 S   1.0  0.0 191:39.65 kswapd0
    9 root      20   0       0      0      0 S   0.3  0.0 329:15.73 rcu_sched
 1024 root      20   0  121220  15916   3264 S   0.3  0.0 364:46.79 google_osconfig
 3390 root      20   0  160044   2312   1536 R   0.3  0.0   0:00.32 top
    1 root      20   0  191164   3124   1652 S   0.0  0.0   1:24.79 systemd
    2 root      20   0       0      0      0 S   0.0  0.0  10:48.88 kthreadd
    4 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    6 root      20   0       0      0      0 S   0.0  0.0  21:36.47 ksoftirqd/0
2

What version are you on?
What is in the logs?
What about hot threads?

3

Version -- 7.6.2

Logs:
Is there a specific log you are looking for or any stack trace that you expect that may cause the CPU spike because I don't see any logs that point to any issues or are there any errors. Sometimes i see "failed to execute bulk item (index)" for a few indexes but that doesnt create any issues in the cluster or nodes(as per my understanding).

Is there way i can upload the hot threads out put, it is a big log I some how cant attach it here.

Thanks for your response @warkolm

4

Any/all logs can be relevant, even if you aren't spotting anything.

Use gist/pastebin/etc.

6
7

@warkolm I have uploaded the cluster details as well in the comments fyi. Please let me know if you need any further details that can help further check on it.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.