Hey All,
What is the best practice to create a number of indexes if I have around 500 to 700 millions of record per day? One index per day or multiple indexes per day (Let's say one per 6 hours) where there is an aggregation need to be done every minute on this data.
Query performance tend to vary with shard size. I would therefore recommend running your aggregations against a single shard to see what a good size is for your use case. In order to keep your shard size below this level, you don't necessarily need multiple indices per day as you can simply increase the number of shards per index to get to the right size. It may be worthwhile looking at this video from Elastic{ON} 2016 which discusses benchmarking and cluster sizing.
Thank you very much!