Can I disable the sorting feature in Elasticsearch's aggregation

As In my case, the bucket size can be as long as millions, and end-user just wants the bucket size instead of the bucket contents. As ES will always return the bucket ordered by doc_count,this content is not I want and very time-consuming.
I've already used the response filtering to just receive the bucket size, but still the query is too slow. Can I just turn off the sorting in the ES's aggregation and speed up my query?