I've read that the best practice for querying/searching is to use a coordinator node, which makes sense to me (a node that's not busy on disk operations for indexing and uses memory for gather phase of searches).
However, when indexing data the best practice is to write directly to data nodes. As far as I know, when a write/index request is issued to a data node, it routes the request to the node which holds the primary shard of the requested index (not an HTTP redirect, the TCP connection is still open to that node but the node forwards the data to the primary shard holder).
In that case, why not use a coordinator node as well? The larger my cluster is, the higher the chance I won't be hitting the node which holds the primary shard and in that case, the data node is going to be busy forwarding all of the data instead of other tasks. Why not create a set of dedicated nodes (maybe even with a load balancer) which will forward the traffic to the right data node?
If I got the explanations completely wrong, could you please explain what's the motivation (in low technical level) for using coordinator node for searches only?
Thanks in advance!