Yep, because you're basically doing N gets, each of which take time to run and then be collated. And the overall response is only going to be as fast as the slowest one.
At a guess, that could be a cache miss. I am not sure you can do much about those tbh.
Any
_searchrequest can be profiled by adding a top-levelprofileparameter: