Hi we are observing too much cpu on cluster suddenly and the hot threads shows below only on management thread pool.
-
hotthreads
99.8% (499ms out of 500ms) cpu usage by thread 'elasticsearch[node1][management][T#2]' -
threadpool stats
node_name name active queue rejected
elstst04 management 4 0 0
elstst02 management 3 0 0
elstst01 management 1 0 0
elstst03 management 5 1 0
elstst05 management 5 5 0 -
Elasticsearch version : 6.3
-
Java 1.8
-
5 Node cluster with 4 CPU and 64gb Memory and 1.5TB of indexes across cluster
-
Old CMS gc collection, timeout seen on node/mon_stats, stopped the monitoring still same.
-
No pending tasks
-
Tasks are mostly on this only
"indices:admin/seq_no/global_checkpoint_sync",
"cluster:monitor/nodes/stats[n]",
Cluster is mostly idle no bulk loads happening but we still see 100% cpu on all data nodes.
Please let us know what all need to be verified