If I understand the problem correctly this is what I call the "disappearing checkboxes" problem.
When you have multiple filtering dimensions to contend with e.g. brand, size and colour my advice is to group user selections into separate bool clauses as follows:
must : [
query-string : [user input],
The min_should_match clause should be one less than the number of dimensions (in this case size/colour/brand). This ensures matches that hit a dimension in the agg tree always satisfy all other category selections.
While this fixes the aggregation results it does mean that hits may contain docs that match only 2 out of 3 top-level filter clauses. The top results are likely to be the ones that tick all the boxes but you can see the problem. To fix that you'd need to add a
post_filter for the hits that insists on all 3 out of 3 filter clauses matching.
This is a general pattern that should work regardless of how many filtering dimensions you have.