Hitting 'too_many_clauses' exception when used in aggregation filter context

I have a terms query returning over 5k clauses and when this is used within the filter context of the query, I never hit this 'too_many_clauses exception - you have exceeded 1024 clauses'.

However if I use this terms query within the aggregation, I am hitting 'too_many_clauses exception'. Is there a way to get past this?

aggs:{
"filter": 
      {
        "bool": {
          "must": [
            {
              "terms": {

              -- 
              }
            }
          ]