How to filter a subset of data within a larger set

Elastic Stack Elasticsearch
1

We have a need to filter data within a larger response in a way that doesn't really feel natural to filters we've used so far, but it seems possible.

Our goal (in words): Only give me records that belong to this 'category_id', that match these 'brand_ids'. This would be simple if we only needed this in the response, however, we need this inside of our default response alongside tons of other categories and brands. The whole idea is blacklisting an entire group of things, then whitelisting a select few to be returned with everything else. Our best attempt was the following:

{
  "not": {
    "and": [
      {
        "terms": []
      },
      {
        "not": {
          "terms": []
        }
      }
    ]
  }
}

But doesn't seem to be working as intended, is something like this possible?

Thanks in advance.