When a shard is held on only one node (ie one primary, no replicas) any caching (eg file-system level or segment level) will make returning queries faster.
When you have replicas your queries will be randomized across replicas, some of which will have empty caches for your query and others warmed.
You can use a client session ID as the custom value in the preference setting to route queries back to the same replica where possible to gain the benefits of a warm cache