If you don't ask for any source manipulation then Elasticsearch treats the document as an opaque sequence of bytes, which it can handle very efficiently. If it has to manipulate the source at all then it must parse it, convert it into a tree of freshly-created objects, exclude the bits you want excluding, and then convert it back into a sequence of bytes for further processing. This extra work can be quite significant.
Note that the source is a stored field, but you are not preloading the stored fields file. Also note that preloading is only a best-effort process and does not guarantee that this data remains in RAM.
An alternative would be to store the field(s) that you do want for these queries rather than parsing them from the source each time.
Another alternative is to exclude this field at index time. This has downsides, of course (no reindexing, no updates, etc.) but maybe that's ok for you.