Thread_pool write change size of queue

[Cristian1988]

Hi,

We have a clúster running with elastic version 7.8.2, that by default the write queue_size of thread_pool is 200

In the lastest versions of elasticsearch this queue_size by default is 10000. My question is can i up this size 200 to 10000?

Sorry for my inglish.!

[Thibault_Richard (Thibault)]

Hi,

Yes, you can set it as you want. ECK does not manage this settings (settings managed by ECK).

In this fantastic post from @jasontedor, he runs through why increasing/changing your threadpool settings may not be the best idea, even if it looks like it solves a problem.

Why is your queue filling up? Because your producers are adding work items to the queue faster than the consumers can take them off the queue.

What would happen if you increase the queue size? The producers will still be producing at the rate they are currently producing, and the consumers will still be consuming at the rate they are currently consuming. This means that the queue will still fill up. Except now, the queue is bigger which will put more memory pressure on the system. In fact, this means the consumers might slow down, which might exacerbate the production rate greater than consumption rate problem. From there, a vicious feedback loop will form.

Queues are helpful for handling variable load. They are not helpful for handing scenarios where the production rate exceeds the consumption rate. In this scenario, a queue filling up is a form of back pressure. It is telling you that you need to slow down the rate of production (or speed up the rate of consumption).

Only increase the queue size if you are in a situation where you have variable load that sometimes exceeds capacity. Do not increase the queue size if you are in a situation where production rate exceeds consumption rate.

Note - This post is a few years old and threadpools have had improvements over time, including the dynamic sizing mentioned above and in the docs. However we still maintain the same advice if you encounter this error/response.

For more questions about this setting, as it is unrelated to ECK, please use the Elasticsearch section of the forum.

[Cristian1988]

Thank you