Min_doc_count Not working with Cardinality aggregate

I am trying to get a distinct list of user ids out of ElasticSearch where the user id was in more than one document across about 100M records.

Min_doc_count works fine if you are doing normal aggregate counts however if you change the aggregate to a cardinality aggregation, the Min_doc_count applies only to the total doc count that make up the aggregate and not the total unique counts within the aggregate.

Has anyone come up with a workaround for this?