Setting thread_pool.index.size in es 5

Elastic Stack Elasticsearch
1

Hi, I started using elasticsearch 5.0.1
I want to increase my index or bulk thread pool size in settings but i get the error Failed to parse value [..] for setting [thread_pool.index.size] must be <= ....;

In documentation (Thread pools | Elasticsearch Guide [8.11] | Elastic) I see that:

Index
For index/delete operations. Thread pool type is fixed with a size of # of available processors, queue_size of 200. The maximum size for this pool is 1 + # of available processors.

But the problem is, elasticsearch is heavily queueing my bulk operations. In previous versions(ex:2.1.1) I could increase this limit and the problem was being solved, indexing was pretty fast and no queue overloading. But even if elasticsearch is using its own implementation of queue, it does not seem to work better with limiting thread size. Now I cannot increase limit and use fast indexing because elasticsearch forces threadpool to a limit.

Is there a way to increase this limit? In previous versions I could increase the limit and os was managing its own thread queue better then elasticsearch. This may depend my data size, network, cpu performance etc. but, I should be able to specify this limit nonetheless.

It seems like the only way to do is to override number of processors that elasticsearch is using. If there a better way?

2

Your solution was wrong because the diagnosis was wrong.

Bulk thread pools are automatically adjusted in ES 5, there is no longer a need to mess with it. They are so large that all CPU cores can be kept busy all the time. The queue_size for bulk indexing is so large that with a bulk request size of 10MB and 200 entries, the heap may get occupied up to by 2 GB just for bulk requests.

The statement os was managing its own thread queue better then elasticsearch makes no sense since effectively it is always the OS that is handling threads.

The new ES 5 default set up tries hard to protect the system from being overloaded by users and becoming unresponsive just because of excessive configuration values. I assume there are other problems in your set up that prevents ES from fast indexing and I suggest to dig deeper before applying the wrong medicine.

1 Like
3

Thank you for the answer.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.