Node thread_pools choked

Elastic Stack Elasticsearch
1

We are using elasticsearch 7.17.6 in one our cluster.
Recently we saw that two of the hot nodes had their thread_pools choked. Increased Queue and rejections.

image
image800×385 63.4 KB

From jstack it seems that the node's CPU is occupied doing merges. Jstack shows threads are runnable however due to merges consuming CPU write, search etc thread_pools are not able to schedule work, creating a queue and hence rejections.

Jstack link - choked_node_jstack.txt · GitHub

Merges stats of this node:

image
image800×450 70.7 KB

To resolve this we eventually had to restart both nodes. Could someone suggest what can/should be done to prevent this in future?

2

Hi, this looks like merge threads eating up CPU and starving your write/search pools. Try lowering index.merge.scheduler.max_thread_count to 1 on the affected indices. Elasticsearch Guide [7.17]

PUT /my_sample_index/_settings
{
  "index.merge.scheduler.max_thread_count": 1
}
3

What type of storage are you using on the hot nodes in question? Is it local SSDs or maybe some type of networked storage?

As you are seeing a lot of throttling it is quite possible that the performance of your storage is not sufficient to support your workload. In a hot-warm architecture where the hot nodes handle all I/O intensive indexing and a lot of the querying of recent data it is vital to have high performance storage.