I have a multi-level aggregation query - that results nested buckets. In order to retrieve all the results I am currently using the the total doc hits number and then divide that by partition_size = 1000, to get the results hashed across x partitions ( x= total[hits]/partition_size) and retrieving the the aggregation results from each partition id. We are using the include clause to use the partitioning approach.
We are using what we think is a safe approach by using total[hits] to derive the total number of partitions, by using a fixed number of partition_size. This is because we do not know if there is a way to accurately know before hand how many aggregation buckets are in the result.
Is there a better approach ?