Using "partial" terms aggregation

I have populate a test index with the following data:

/test/_bulk
{"index": {}}
{"types": [1, 2, 4]}
{"index": {}}
{"types": [3]}
{"index": {}}
{"types": [1, 4]}

On that index I'm trying to a build a query where I filter on the types field while matching the name field with a query string.

The query I built works fine but I'd like to use terms aggregation on the types field to display the result count per type on a facet.

When using aggregation like so:

"aggs": {
    "types_count": {
        "terms": {
            "field": "types"
        }
    }
}

My problem is that the bucket I get only contains the types that where present in the filter of my query.

I searched for Partial aggregation But only found Global aggregation that exclude the whole query from the bucket content.

Is there any way to exclude a part of a query to build the aggregation bucket or will I need to build a second query that does not use filtering ?

Thanks