That fields parameter isn't for getting things out of the _source. It does so to be backwards compatible with Elasticsearch 0.90. I believe it won't fetch from the source in 5.0 but I could be mistaken.
Your workaround works but it won't work forever. Your comment about JSON is mostly right (technically they could be serialized as YAML or SMILE or CBOR as well). My suggestion is if you are using sourceAsMap() to yank numbers out of the map and cast them to Number and call longValue/floatValue/intValue. We've already boxed them to cram them into the map so there is no extra cost there.
The reason they come back in JSON is that Elasticsearch documents don't keep their Java class information. source(Map) is entirely an API level construct - it immediately serializes the map to a byte array. Those object never make it over the wire to Elasticsearch and even if they did it wouldn't know what to do with them.
Further, unless it has to, Elasticsearch doesn't parse the _source when it returns documents. This is a fairly significant speed up. It only has to do that parsing if you have source filtering or if you ask for the response to be in a format that differs from the one that you indexed the documents with. It figures out the format that the documents are in by sniffing the first few bytes.