Aggregating on docvalue_field from inner_hits

[Jeff_Potts]

I have a query that includes a nested query with inner_hits enabled. I am using docvalue_fields within inner_hits to expose the value from within the nested object that I want to aggregate on:

{
  "query": {
    "bool": {
      "must": [
        {
          "nested": {
            "path": "permissions.permissions",
            "query": {
              "bool": {
                "must": [
                  {
                    "term": {
                      "permissions.permissions.authorityType.keyword": {
                        "value": "GROUP"
                      }
                    }
                  },
                  {
                    "term": {
                      "permissions.permissions.inherited": {
                        "value": false
                      }
                    }
                  }
                ],
                "must_not": [
                  {
                    "wildcard": {
                      "permissions.permissions.authority.keyword": {
                        "value": "GROUP_site_*_Site*"
                      }
                    }
                  }
                ]
              }
            },
            "inner_hits": {
              "_source": false,
              "docvalue_fields": [
                "permissions.permissions.authority.keyword"
              ]
            }
          }
        }
      ],
      "must_not": [
        {
          "term": {
            "siteId": {
              "value": "rm"
            }
          }
        }
      ]
    }
  }
}

The query works great. What I need now is a list of the unique values for permissions.permissions.authority.keyword across all my hits. How do I write an aggregation that aggregates on the inner_hits docvalue field? I've tried nested aggregations and non-nested aggregations but I must not have my paths correct or something because I am not even getting a list of buckets.