Hello, I am running an ES 7.6.0 cluster and once in a while I need to perform a full index rebuild.
Currently I need to index 15 million documents and my indexer currently is able to be fine tuned on request size, maximum concurrent calls, delay etc. All index calls happens via bulk insert.
What I am experiencing is no errors on any bulk response but still in the end I have missing documents.
The only way I found to index everything safely was to dramatically reduce both payload size and number of concurrent requests.
I don't think it is feasible for me to use a sort of binary search approach to find the best settings for such task, because no matter what I do the optimal configuration is going to change with an increasing amount of documents.
What is the best way to manage the request volume performed? Is there any way my indexer can be aware of the cluster's condition while operating?
Thanks in advance!