my Elasticsearch consists of 1master and 3data nodes, and it is not real-time based data stored. we are inserting bulk data once per day (1,000,000 lines), and stored indices size is about 500GB. most index data are being updated so it is persistent size.
there are 60 side nodes (out of Elastic) which generating about 10,000(TPS) SEARCH queries. the cluster has 2 indices and total volume is about 500GB as i mentioned.
we are observing "EsRejectedExcutionException" on the cluster. below is a sample of the error.
"EsRejectedExcutionException: rejected execution of org.elasticsearch.transport.TcpTransport$RequestHandler@442a9767 org.elasticsearch.common.util.concurrent.EsThreadPoolExcutor@36ff918[Running,pool size=49, queued tasks=999, completed tasks=22539068943]]"
i researched the thread pool size is set as 1000 as default, and changing this size is not really recomendable.
to resolve this, is there anything i need to check on the cluster? please advise what i should invstigate on my cluster side. i know adding nodes are strong recommendation but would understand if there are anything i can do with the current cluster.