With more an increasing number of shards and nodes, there is more data that will need to be moved between the nodes, and it may be that this slows down the latency. I would however accept you to be able to handle a higher concurrent query throughput with the larger cluster.
Benchmark with as realistic load as you can (type and volume). You can tune the cluster differently depending on whether you are looking for low latency of few concurrent queries compared to very high number of concurrent queries.