I have 3 nodes setup with default 5 shards and 1 replica. Now I am indexing 1.3TB daily. I have read that shard should atleast max at 50GB now I am wondering how many shards do I need for this kind of setup? Does 10 primary and 10 replica enough? Or 20 primary and 20 replica? Also 1.3TB are both primary (600GB) and replica (600GB). Thanks!
The rule of thumb for shard size is no more that 50G per shard, so you're going to probably need closer to 30.
Also, the number of replicas you specify is per index, so 1 or 2.
You mean to say 30 primary shard? Or 30 shards total? Also how can I increase the number of shard? Just put number_of_shard: 30 in elasticsearch.yml?
May I suggest you look at the following resources about sizing:
https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing
And https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right
1 Like
Thanks a lot. I will give it a time to read. Meanwhile, I want to experiment the number of shards since It's taking awhile rerouting shard from one node to another since I have atleast 150g per shard.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.