The coordinating node is the one hit by the search request. By default, every node can connect to remote clusters, hence the coordinating node will talk directly with the remote cluster.
In case you configured some nodes only to connect to remote clusters, through the
cluster.remote.connect setting, then Kibana needs to send CCS requests directly to one of those nodes or it will get back an error as the coordinating node cannot connect to remote clusters.
When it comes to the remote datacenter, by default three gateway nodes are selected when a remote cluster is registered (can be increased or decreased through
cluster.remote.connections_per_cluster) and those are the only nodes that will receive incoming requests from the local datacenter. In case the request needs to go to other nodes that hold the relevant shards, the gateway nodes will forward such requests to the relevant nodes. In your picture, federation nodes are by default data nodes too, so not every request goes necessarily from the federation node to the data node, unless the selected federation nodes (the selection can be controlled through node attributes) don't hold data.
The master elected node may come into picture if it gets selected to receive CCS requests, though keep in mind that if you have dedicated masters, they are left out of the selection starting from 6.4.0, meaning that dedicated masters won't be used as gateway nodes.
Does this clarify things? Do you have any further questions?