Hello,
I've been running a 50 data node ES cluster with 1.7TB of daily indices for quite some time now. I've just deploy a new 50 node ES 7.0 cluster and was wondering, given the daily index size of 1.7TB with 30 day retention and growing, what is the recommend number of shards and replicas that will perform well for such large indices. According to the documentation the size of a single shard should not be bigger than 30GB, which means that I will need way more than 40 shards, thus more data nodes.
Any suggestions are welcome!