Having read-only shard greater than 50 GB

I recently re-indexed past 18 months of day-wise indices (18 months *30 days = 540 indices with 1 primary shard per index) into 18 monthly indices (18 indices with 12 primary shards per index).

No indexing happens on these past indices. They are effectively read-only.

The reason I arrived as 12 was based on calculations that the average of sum of day-wise-indices per month came to be around 525 GB. So I opted for 12 indices so that each shard is less than 50 GB.

Since I relied on average and created a template accordingly, some monthly indices actually had data of about 700 or 800 GB. With 12 shards, per shard it comes to be 67 GB.

My question is: Is it okay to have read-only shards that are greater than 50 GB like 70 GB? Or that for those months, I should increase the number of shards to 50?

Since all these indices are read-only, I'm planning to deploy best_compression and forcermerge to 1 segments on them.

Will appreciate some inputs on this.

Yeah, that should be ok.

If you're talking 100GB+ shards it becomes harder to manage.

1 Like

Thank you very much.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.