Kibana field_stats request takes 20000 ms if number of Indices and shards are more

I've recently opened an issue on Github to dig into this (https://github.com/elastic/kibana/issues/9386) .. we're seeing the same issue, but its upwards of 35s to return the query. I strongly suspect that its our "warm" data nodes that hold the majority of our data, but is old "closed" indexes. I think that they're slower to respond, and this query requires all the nodes in the cluster to respond it seems.