Conditional Sorting based on specific field values

I am working on optimizing the sorting of our search results, especially the sorting by Name (A-Z), Name (Z-A).

This is the relevant part of the mapping:

 "roles": {
      "type": "nested",
      "properties": {
        "role_type_id": {
          "type": "keyword"
        },
        "inverted_name": {
          "normalizer": "names",
          "type": "keyword"
        }
      }
    }

The roles.role_type_id can be either A1, A2, B1, B2, C1 ... etc. Only the A1 an B1 types are relevant to the actual sorting.

The current sorting looks like this in the search request:

"sort": [
    {
      "roles.inverted_name": {
        "order": "desc",
        "nested": {
          "path": "roles",
          "filter": {
            "terms": {
              "roles.role_type_id": [
                "A01",
                "B01"
              ]
            }
          }
        }
      }
    }
]

The problem is that documents can have only A1, only B1 or both A1 and B1. (There are no documents that have neither A1 or B1 ids, one of them is always required). Our setup works fine for cases where there is only one of either present in a document. But in the cases were both values exist, i want to prioritize the values in roles.inverted_name that have the roles.role_type_id A1.

For example:

{"roles.name_inverted": "miller, xyz", "roles.role_type_id": "A1"}
 {"roles.name_inverted": "baker, xyz", "roles.role_type_id": "B1"}

If we sort this with out current setup, the value used for sorting from A-Z is "baker", and by sorting from Z-A it's "miller".

A-Z
baker

Z-A
miller

I need it to be "miller" in both cases, though.

A-Z
miller

Z-A
miller

Any ideas how i can achieve that?