Generally the top level query is more efficient than the filter agg. The filter agg is really for if you need an extra filter inside of another agg or something like that.
Your right that the aggs don't do scoring, but if you don't fetch any documents then the top level query doesn't use the score either. If we don't need the score we don't build it.