Query Boosts don't seem to correctly affect Aggregations?

I have a scenario where I am using match queries boosting various terms. This in the normal search sense has a great result of limiting the results. However the boost doesn't have the same affect when used in an aggregation (such as a date histogram). The results are very close to when no boost is used and many orders of magnitude off using boost in a normal search. Is there a way to get the boost to correctly affect the aggregation bucket counts?

A boost affects a document's score and its relevancy amongst the documents
returned. An aggregation is computed amongst all the documents returned,
regardless of their score. If a document is returned by a query, it will be
in the aggregation, no matter what score it has.

Perhaps you need an aggregation that works only on the top-n documents. The
sampler aggregation released with 2.0 might help you achieve that:
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-sampler-aggregation.html

Ivan