Basically, yes.
In ES 5.x, numerics use a new datastructure (BKD tree). This allows better compression, faster numeric operations and lower memory usage... but it is not ideal for "point lookups" like a term query. E.g. it is designed for numeric style operations like ranges, but not single value lookups.
If that field is only used for exact-match lookups, you can re-index it as a keyword. Keyword fields are optimized for exact-match lookups and will be a lot faster.
More info: Tune for search speed | Elasticsearch Guide [master] | Elastic
To dive a bit more into technicals, the BKD datastructure doesn't support sorted iteration, so it has to collect all matches, sort the array and then return an iterator to that sorted array (paraphrasing). That process happens during the build_scorer step. This process isn't bad when dealing with numeric ranges since the cost is amortized over all the values that are being iterated over, but can get expensive when asking for a bunch of individual points.
2.x didn't have BKD trees, hence the difference in performance.