Hi we are observing too much cpu on cluster suddenly and the hot threads shows below only on management thread pool.
99.8% (499ms out of 500ms) cpu usage by thread 'elasticsearch[node1][management][T#2]'
node_name name active queue rejected
elstst04 management 4 0 0
elstst02 management 3 0 0
elstst01 management 1 0 0
elstst03 management 5 1 0
elstst05 management 5 5 0
Elasticsearch version : 6.3
5 Node cluster with 4 CPU and 64gb Memory and 1.5TB of indexes across cluster
Old CMS gc collection, timeout seen on node/mon_stats, stopped the monitoring still same.
No pending tasks
Tasks are mostly on this only
Cluster is mostly idle no bulk loads happening but we still see 100% cpu on all data nodes.
Please let us know what all need to be verified