REST client initialization

I would take out only dedicated masters, it is better not to load them with requests as their goal is to be there to potentially become the master once needed (also one of them will be the master).

You could send searches only to coordinating nodes, then they will proxy the requests and also handle the reduce part of the search but all your requests will always have to be forwarded to the data nodes where the data sits. This may be beneficial but I would not do it to start with, I would try to test the benefits of such a setup first.

We have been working on a new feature (coming with 6.4) that will allow to have custom node selectors. With that it will be possible to have for instance an allocation aware selector, which tries to send request to the closest nodes. I am not sure about sending indexing to data nodes and search to coordinating nodes only, I would test it before assuming it's beneficial. The only way to do that without the node selector improvement is though to have two separate client instances that point to two different set of nodes.

Cheers
Luca