Query for specific traits in a household

[marksparrish (Mark Parrish)]

I am trying to retrieve all of the household that have only females in it.
My document structure is -

name_first: keyword
name_middle: keyword
name_last: keyword
physical_address_id: keyword
gender: keyword

Gender is M,F,O

I can get the aggregation with

{
"size" : 0,
"_source" : false,
"stored_fields" : "_none_",
"aggregations" : {
  "groupby" : {
   "composite" : {
    "size" : 1000,
    "sources" : [
      {
        "769" : {
          "terms" : {
            "field" : "physical_id.keyword",
            "missing_bucket" : true,
            "order" : "asc"
          }
        }
      },
      {
        "746" : {
          "terms" : {
            "field" : "gender.keyword",
            "missing_bucket" : true,
            "order" : "asc"
          }
        }
      }
    ]
  }
}
}
}

This is where I'm stuck. Post Filter does not see to fit or at least I can't get it to work.