About a year ago, lots of people from Elastic were singing the praises of the field_stats_api, specifically how it can be used to figure out which indices to target a search on. Kibana still uses this mechanism in the background. I came to implement something similar myself yesterday, and found now that it has been deprecated and to use the field_capabilites API instead, although this doesn't (as far as I can tell), support that one feature of the field_stats_api that I wanted. It suggested in the documentation that aggregations can be used instead, but in my scenario, I have too many shards to search against. So I'm curious as to how Kibana is going to change in a future release to cope with the loss of the field_stats_api?
Here is the problem:
For a given index pattern 'daily_index-*' we have 30 days of indices. Lets assume that each index has 50 shards, which gives a total of 1500 shards. For a given time range, which could span multiple days, I want to know which indices contain data for that time range. The field_stats_api was able to tell you very quickly, which indices contained data spanning the time range in question (current Kibana implementation).
An aggregated search against the timestamp will fail (unless I change some limits set in Elasticsearch), as it is hitting more that 1000 shards. The field capabilities API doesn't seem to have the same function. So apart from manually figuring out the index names, how can I get ES to efficiently tell me which indices contain the data I need to search against? I'm sure the clever people writing ES and Kibana, have got a new way to efficiently do this and I'm keen to learn how.
Thanks in advance.