The terms aggregation has a min_doc_count setting which does this but can be inaccurate if there are high numbers of ids that occur infrequently. It’s what I call the Elizabeth Taylor problem
If you have many unique rare terms you may need to look at a different strategy which we can discuss.