ESRejectedExecutionException using _msearch

Hey, the rejection failures means that you have overloaded the search thread pool in execution concurrent search requests.

The search thread pool defaults to ~2 the number of cores you have, and 1000 items in the queue. When there are more, those will be rejected. Each execution is a shard level execution (so a search on 5 shards, on a single node, will require 5 concurrent threads).

This mechanism is important not to change, since it helps creating back pressure when the node is overloaded, instead of overloading it more.

The multi search API simply goes and executes all the search requests concurrently. So if you have 10 search requests there, hitting all indices, it will concurrently hit all indices and shards, which might cause the data nodes to start rejecting requests since they are being overloaded concurrently.

I hope my explanation makes sense. If you need the msearch execution to work, across all the indices and shards you have, you probably need a few more nodes to make sure you have the execution capacity to execute it. Another way is to execute each search in a serial manner not using msearch, which will add to the latency of the execution.