That actually makes sense and I think it confirms what I was thinking. The problem is that in newer versions of ES we use this IndexOrDocValues from Lucene which decides to execute the query either using the index or docValues, depending in some heuristics.
In your case, you probably are storing quite big arrays in the sums field which is screwing up those heuristics and we are taking a bad decision on how to execute the query.
One way to overcome this is to index that field without docValues. The side effect is that you cannot use that field for sorting or aggregations.
Meanwhile I am opening an issue in Lucene to improve such heuristics.