Index threadpool vs write threadpool

Elastic Stack Elasticsearch
1

According to https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-threadpool.html, a node holds:

index threadpool
For index/delete operations. Thread pool type is fixed with a size of # of available processors , queue_size of 200 . The maximum size for this pool is 1 + # of available processors .

write threadpool
For single-document index/delete/update and bulk requests. Thread pool type is fixed with a size of # of available processors , queue_size of 200 . The maximum size for this pool is 1 + # of available processors .

Anyone can explain what's the difference?

2

Hi Bli,

In 6.x the WRITE threadpool is used for shard-level bulk and resync replication operations. The INDEX threadpool is used for other update, delete, and indexing operations.

3

Thank you @dakrone.

We use bulk API to write to ES. That worked perfect for certain types of data with large volume and high throughput is desired (e.g. metrics). For other types of data (e.g. inventory), we want them to be searchable near real-time. Since for now all our data use the same bulk API, they end up with competing resource and as result sometimes inventory data will not be indexed as quick as we'd like.

One way i'm looking to is use index API for inventory data, and bulk API for other types of data. This way they are handled by different threadpool, and likely less competing. Do you think it would work?

However looking as ES source code in master, it looks like index threadpool is deprecated? Then the above solution might work in 6.3, it won't work in the future.

Really appreciate your help!

4

Hi Bli,

One way i'm looking to is use index API for inventory data, and bulk API for other types of data. This way they are handled by different threadpool, and likely less competing. Do you think it would work?

In order to do this you would have to write a plugin that changed these APIs, instead I'd recommend sticking with the vanilla threadpooling of ES.

However looking as ES source code in master, it looks like index threadpool is deprecated? Then the above solution might work in 6.3, it won't work in the future.

That's correct, in 7.0 there is only the WRITE threadpool, so I would not count on this.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.