EsRejectedExcutionException error

[c8868308c0cfc8cf35cc (jason_)]

hello,

my Elasticsearch consists of 1master and 3data nodes, and it is not real-time based data stored. we are inserting bulk data once per day (1,000,000 lines), and stored indices size is about 500GB. most index data are being updated so it is persistent size.

there are 60 side nodes (out of Elastic) which generating about 10,000(TPS) SEARCH queries. the cluster has 2 indices and total volume is about 500GB as i mentioned.

we are observing "EsRejectedExcutionException" on the cluster. below is a sample of the error.

"EsRejectedExcutionException: rejected execution of org.elasticsearch.transport.TcpTransport$RequestHandler@442a9767 org.elasticsearch.common.util.concurrent.EsThreadPoolExcutor@36ff918[Running,pool size=49, queued tasks=999, completed tasks=22539068943]]"

i researched the thread pool size is set as 1000 as default, and changing this size is not really recomendable.

to resolve this, is there anything i need to check on the cluster? please advise what i should invstigate on my cluster side. i know adding nodes are strong recommendation but would understand if there are anything i can do with the current cluster.

thank you!

[Christian_Dahlqvist]

It sounds like you might be overloading the cluster given the amount of data, updates and queries. Do you have monitoring installed so you can see what its limiting performance in the cluster? Is it CPU bound? Is it disk I/O that is limiting performance?

If you provide more information about your data and how you are querying it, someone might be able to help you tune it, but based on the information provided I would guess the cluster may need to be scaled up/.out.

[c8868308c0cfc8cf35cc (jason_)]

Thank you for your advice. Okay i will provide you more details. Thanks!