Write thread pool queue size defaults

Elastic Stack Elasticsearch
1

Hello, everyone!

I'm tuning ES 6.8 for indexing performance and noticed one interesting thing. There is a significant difference in the default value of the q size in versions 6.8 and 7.x. The documentation states:
ES 6.8
write
For single-document index/delete/update and bulk requests. Thread pool type is fixed with a size of # of available processors , queue_size of 200 . The maximum size for this pool is 1 + # of available processors .
ES 7.x
write
For single-document index/delete/update and bulk requests. Thread pool type is fixed with a size of # of allocated processors , queue_size of 10000 . The maximum size for this pool is 1 + # of allocated processors .

So, that's 10,000 vs 200.

I'm wondering , would it be recommended to change this value to 10,000 in my version 6.8 cluster and would it improve the situation with 429 errors, keeping in mind that I use separate coordinator nodes and the cluster size is really huge (100+ nodes)

2

No. The longer queue in 7.x makes sense because 7.x limits the queue size in other ways, and typically won't get near to 10,000 items before hitting those other limits. Increasing the queue length in 6.8 to 10,000 risks replacing the 429 errors you experience today with OutOfMemoryError failures.

1 Like
3

Thanks, David! Will check the other options then.
BTW, could you recommend a knowledge source about bulk rejections, aside from this one? Why am I seeing bulk rejections in my Elasticsearch cluster? | Elastic Blog
We are planning to migrate to 7.x, would it help to reduce the amount of them?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.