We are using Elasticsearch 5.6 (Next week we will upgrade to 7.6) for our application, we are using elasticsearch mostly for bulk indexing and much fewer for querying the data.
Our cluster contains 8 nodes. 1 Data node and the others are both data and master eligible.
Each node has 8G RAM (4G from it for heap usage) and 2CPU.
We have 2 daily indexes, daily log index with two shards (Around 500k-1mil documents per day) and daily data index with two shards (Around 4-5 mil documents per day).
Lately we get lot of rejected from BULK and SEARCH thread pools we tried to increase our nodes from 6 to 8 but we still see some rejected operations even it did reduce the rejected rate.
Each node doesn't use high amount of CPU or heap but sometimes the load is going high.
We don't know what is the best way to solve those errors or how to identified the cause.. Increase shard size/ stronger machines/ split to dedicated master and data nodes..
How can we fix the rejected?