EsRejectedExecutionException - Need suggestions to overcome

Elastic Stack Elasticsearch
1

Hi Team,

In our production env cluster we are getting EsRejectedExecutionException for some requests.
We have observed that there is a recurring thread running by a particular customer which are doing indexing and occupying the thread pool for some time
We have confirmed this in indexing log and trying to stop the thread.
mean while is there any other tweak or suggestion to overcome this, As it affects the customer who are doing some routing requests

We can increase the heap to some +1/2 GB for a node in a 5 data node cluster
here we don't have resource for a new data node addition

Below are the config
8 node cluster - 5 data node + 3 master (dedicated)

5 vm's where 5 data nodes are deployed with 10vcpu in 2 vm and 6vcpu in 3 vm

we also want to know how the thread pool will work and what is the difference b/w "available_processors" and "allocated_processors"

we are using ES 7.3.2 and java 8 and cms gc

2

These things are all very old and now unsupported. Newer versions have much better protection against this kind of rejection. The first thing to try is to upgrade to a supported version, at least 7.17.x.

3

is there any possibility to avoid this in lower version 7.3.2

4

I don't really remember how to even start troubleshooting this kind of thing in such an old version, it's been so long (almost 5 years).

5

Are you using tiering or are all data nodes part of a single tier? Elasticsearch assumes all data nodes in a tier are equal with respect to resources, so the fact that some of your nodes have less resources could be an issue, especially is such an old version, and potentially also result in different thread pool sizes for the different nodes.

Indexing queues can build up if indexing is slow, and this can often be caused by slow storage, so that may also be worthwhile looking into. I would also recommend you have a look at these docs and potentially also this rather old blog post.