We are using elasticsearch 7.17.6 in one our cluster.
Recently we saw that two of the hot nodes had their thread_pools choked. Increased Queue and rejections.
From jstack it seems that the node's CPU is occupied doing merges. Jstack shows threads are runnable however due to merges consuming CPU write, search etc thread_pools are not able to schedule work, creating a queue and hence rejections.
Jstack link - choked_node_jstack.txt · GitHub
Merges stats of this node:
To resolve this we eventually had to restart both nodes. Could someone suggest what can/should be done to prevent this in future?