I recently re-indexed past 18 months of day-wise indices (18 months *30 days = 540 indices with 1 primary shard per index) into 18 monthly indices (18 indices with 12 primary shards per index).
No indexing happens on these past indices. They are effectively read-only.
The reason I arrived as 12 was based on calculations that the average of sum of day-wise-indices per month came to be around 525 GB. So I opted for 12 indices so that each shard is less than 50 GB.
Since I relied on average and created a template accordingly, some monthly indices actually had data of about 700 or 800 GB. With 12 shards, per shard it comes to be 67 GB.
My question is: Is it okay to have read-only shards that are greater than 50 GB like 70 GB? Or that for those months, I should increase the number of shards to 50?
Since all these indices are read-only, I'm planning to deploy best_compression and forcermerge to 1 segments on them.
Will appreciate some inputs on this.