I think disk IOPS was the bottleneck for us. We had some very heavy documents (about 35 - 40MB) in size. Their retrieval was really slow.
What we ended up doing was to disable their storage (nested fields) from the _source field. We didn't really need them to be returned in the response but needed them in the index for some scoring.
Thank you for helping us with this issue. Much appreciated!