Terms aggregation on multiple indexes, ignore text fields

I have multiple indexes, that I'd like to query at the same time, and find out which values are present in some field, with a terms aggregation.
The problem is, this field is sometimes mapped as a keyword, and sometimes as a textfield with a seperate keyword-field (and fieldddata is disabled).
So, when I do a terms aggregation, some indexes succeed, while others gives an error, "text fields are not optimised for ...".

How can I tell elastic to ignore these textfields? I don't really need a full count over all indexes, just get an overall impression. Ideally, I'd be able to do something like:

GET my_index/_search
{
  "query": {
    # Some query
  }
  "aggs": {
    "keyword-direct": {
      "terms": {
        "field": "my_field"
      }
    },
    "keyword-subfield": {
      "terms": {
        "field": "myfield.keyword"
      }
    }
  }
}

This should give me two aggregations, one with the results from the indices that have my_field as a keyword, and one with results for the indices that have a my_field.keyword-subfield.

However, the fact that the first aggregation fails causes the whole shard to just return an error, and I don't get the other results from that shard (neither the query-results nor the second aggregation).

The only workaround I've found is to split it into two queries. But then I have to manually check if all errors are expected, and also combine the query-results. It feels like a hack, and I'd think it could be done in a cleaner, more robust way.

I also noted that if there are no results for the indexes with textfields, whether or not I get an error is undefined.