Management thread taking too much cpu

[Geeks_Insights]

Hi we are observing too much cpu on cluster suddenly and the hot threads shows below only on management thread pool.

1. hotthreads
   99.8% (499ms out of 500ms) cpu usage by thread 'elasticsearch[node1][management][T#2]'

2. threadpool stats
   node_name name active queue rejected
   elstst04 management 4 0 0
   elstst02 management 3 0 0
   elstst01 management 1 0 0
   elstst03 management 5 1 0
   elstst05 management 5 5 0

3. Elasticsearch version : 6.3

4. Java 1.8

5. 5 Node cluster with 4 CPU and 64gb Memory and 1.5TB of indexes across cluster

6. Old CMS gc collection, timeout seen on node/mon_stats, stopped the monitoring still same.

7. No pending tasks

8. Tasks are mostly on this only
   "indices:admin/seq_no/global_checkpoint_sync",
   "cluster:monitor/nodes/stats[n]",

Cluster is mostly idle no bulk loads happening but we still see 100% cpu on all data nodes.

Please let us know what all need to be verified

[DavidTurner]

@Geeks_Insights

• Elasticsearch version : 6.3

This version was released over two years ago and is long past the end of its supported life There's definitely been many performance improvements and bug fixes in the meantime.

Can you share the full hot threads output?

[Geeks_Insights]

Hi David ,

Thank you for looking into it.

Now the management threads gone and the CPU usage came down. Yesterday for 12 hours almost it was like, even after reboot it was same. There's a bulk load happened on the cluster.

Why and how this management threads used and how it causing spike if you can help me out, will check in next time and get more details.

Thanks