Hi team,
First time posting a topic in here.
I have a 6-node cluster which look like this:
heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
31 92 18 9.53 8.86 8.77 dimr - node-6
26 93 29 15.48 18.94 25.95 dimr * node-5
71 91 25 14.23 17.64 25.82 dimr - node-3
34 71 80 40.09 32.82 23.56 dimr - node-2
68 83 21 11.65 10.55 10.31 dimr - node-4
30 79 92 36.20 30.51 21.76 dimr - node-1
As you can see, 2 nodes have high CPU usage and per my observation it rotates. There's always 2 nodes within the cluster that has high CPU utilization. Usually above 70% and gets as high as 90%+
Based on initial checking, data stream logs-system.security [Fleet-Managed]
has the most indices with more than 22,000/s index rate(based on Stack Monitoring - Indices tab). Default configuration for this is to have 1 primary shard with default rollover of 50gb maximum primary shard size.
Appreciate any help I can get on this issue.
Thanks