Increase search thread pool capacity by adding more non "data" nodes

[ben5556]

Hi,

We encounter search thread pool rejection errors on random days during peak usage. I was wondering if we can dynamically (scale UP) add non "data" nodes for example more master or other type of nodes to increase the search thread pool capacity during peak usage and scale DOWN these nodes when the usage goes down outside business hours ?

The reason behind adding non "data" nodes is to prevent shard re-allocation which is an intensive operation and impact performance.

Is this a good approach ? Would you recommend any alternative approach to reduce the occurrence of search thread pool rejections ?

Thanks!

[spinscale (Alexander Reelsen)]

I expect the thread pool rejections to happen on the node executing the searches, so that approach won't help.

Either spinning up more nodes so share the load, or having nodes with more CPUs as by default the number of threads is based on the number of cores. Last but not least each thread pool can be configured manually but that is not really solving the problem, just symptoms I'd say.

Hope this helps.

[ben5556]

Thanks @spinscale. So to clarify, only the data nodes who hold the actual data can execute searches ?

[spinscale (Alexander Reelsen)]

That is correct. If you hit a non-data node or a master only node, all those nodes do, is forwarding the request to those node holding the shards that should be queried.