Min_doc_count combined with order by average

[Christian_Hansen_Jam (Christian Hansen (James))]

As the topic says, Im having problem sorting data when combining with min_doc_count with an average value.
The data om working with is logfiles from a webserver. Two of the field are url.path and event.duration.
My goal is to identify the slow endpoints.
This is fairly easy and solved that problem with the following query.

"aggs" : {
     "endpoints" : {
         "terms" : { 
           "field" : "url.path",
           "size" : 20,
           "order": {
             "average_responsetime": "desc"
           }
         },
         "aggs": {
           "average_responsetime": {
             "avg": {
               "field": "event.duration"
             }
           }
         }
     }
 }

When looking a the data i realized that the top results were bucket sizes of one or two. This makes sence, since they are not cached and thus has a long response time and are not really that interesting. To get rid of these small buckets i found the min_doc_count and turned it up to 100.

"aggs" : {
        "endpoints" : {
            "terms" : { 
              "field" : "url.path",
              "size" : 20,
              "min_doc_count": 10, 
              "order": {
                "average_responsetime": "desc"
              }
            },
            "aggs": {
              "average_responsetime": {
                "avg": {
                  "field": "event.duration"
                }
              }
            }
        }
}

But this does not work. The only way to get min_doc_count working is by removing

"order": {
    "average_responsetime": "desc"
}

but then I dont get the documents im interested in.
Any ideas on how i can get this to work ?