# Sharding Strategy **URL:** \ **Category:** Elasticsearch\ **Created:** [October 11, 2017, 3:24pm UTC](https://discuss.elastic.co/t/sharding-strategy/103573 "2017-10-11T15:24:49Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![Manikanth\_Reddy](https://avatars.discourse-cdn.com/v4/letter/m/5fc32e/32.png) [@Manikanth\_Reddy](https://discuss.elastic.co/u/Manikanth_Reddy)\ **Post date:** [October 11, 2017, 3:24pm UTC](https://discuss.elastic.co/t/sharding-strategy/103573/1 "2017-10-11T15:24:50Z")
Hi, We have a cluster of 8nodes with 3,248 indices and 29,544 shards with 1 replica. We have 5shards per each index. These are daily indices with size ranging from 10MB to max 2GB. Out of these 8 nodes, 3Master,3Data and 2 client nodes. We are using ELK 5.2.1. I believe that having more shards is impacting Kibana. Could you please suggest me whether it is a good strategy to have 5 shards per index or decreasing would be better? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 11, 2017, 3:29pm UTC](https://discuss.elastic.co/t/sharding-strategy/103573/2 "2017-10-11T15:29:08Z")
That is an excessive amount of shards for a cluster with only 3 data nodes. Please read [this blog post](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster), which contains discussion and guidelines on this topic. ---
**Author:** ![Manikanth\_Reddy](https://avatars.discourse-cdn.com/v4/letter/m/5fc32e/32.png) [@Manikanth\_Reddy](https://discuss.elastic.co/u/Manikanth_Reddy)\ **Post date:** [October 16, 2017, 2:25pm UTC](https://discuss.elastic.co/t/sharding-strategy/103573/3 "2017-10-16T14:25:36Z")
@Christian_Dahlqvist Thanks for the post. As per the post, we have 3 data nodes of 32 gb heap. So I need have only 1 primary shard for each index. But if I make number of shards to 1 for each daily index instead of default 5 shards, how can we handle node failure. If a node or shard fails somehow, are we going to lose the data on that shard. Thanks. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 16, 2017, 2:29pm UTC](https://discuss.elastic.co/t/sharding-strategy/103573/4 "2017-10-16T14:29:57Z")
You handle node failures by having a replica shard configured. The number of primary shards does not really matter. ---
**Author:** ![Manikanth\_Reddy](https://avatars.discourse-cdn.com/v4/letter/m/5fc32e/32.png) [@Manikanth\_Reddy](https://discuss.elastic.co/u/Manikanth_Reddy)\ **Post date:** [October 16, 2017, 2:40pm UTC](https://discuss.elastic.co/t/sharding-strategy/103573/5 "2017-10-16T14:40:24Z")
So, currently we have 1 replica in place. Shall I make it to 2 replica by having 1 shard or Shall I keep the same 1 replica with 3 shards? ---
**Author:** ![Manikanth\_Reddy](https://avatars.discourse-cdn.com/v4/letter/m/5fc32e/32.png) [@Manikanth\_Reddy](https://discuss.elastic.co/u/Manikanth_Reddy)\ **Post date:** [October 16, 2017, 3:26pm UTC](https://discuss.elastic.co/t/sharding-strategy/103573/6 "2017-10-16T15:26:51Z")
@Christian_Dahlqvist Could you also explain why number of primary shards does not matter here. Sorry if I am asking so many questions. Thanks. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 16, 2017, 8:36pm UTC](https://discuss.elastic.co/t/sharding-strategy/103573/7 "2017-10-16T20:36:11Z")
The number of primary shards are important as it affects the total shard count. All primary shards however hold different data. If you want to make sure you can cope with a node going down without data loss, you need to make sure you have at least 2 copies of each shard, which is what replicas give you. ---
**Author:** ![Manikanth\_Reddy](https://avatars.discourse-cdn.com/v4/letter/m/5fc32e/32.png) [@Manikanth\_Reddy](https://discuss.elastic.co/u/Manikanth_Reddy)\ **Post date:** [October 16, 2017, 9:07pm UTC](https://discuss.elastic.co/t/sharding-strategy/103573/8 "2017-10-16T21:07:29Z")
@Christian_Dahlqvist Thanks. Then I will go with 1 primary shard for each daily index along with 2 replicas. Thanks. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 16, 2017, 9:11pm UTC](https://discuss.elastic.co/t/sharding-strategy/103573/9 "2017-10-16T21:11:59Z")
1 replica is often sufficient as it results in 2 copies of each shard. Given the number of indices and shards in your cluster I however suspect you may need to reduce it even further than what you would get just by reducing the primary shard count. I would recommend starting to use weekly or monthly indices for smaller indices with a long retention period. ---
**Author:** ![Manikanth\_Reddy](https://avatars.discourse-cdn.com/v4/letter/m/5fc32e/32.png) [@Manikanth\_Reddy](https://discuss.elastic.co/u/Manikanth_Reddy)\ **Post date:** [October 16, 2017, 9:18pm UTC](https://discuss.elastic.co/t/sharding-strategy/103573/10 "2017-10-16T21:18:56Z")
Sure @Christian_Dahlqvist. Thanks for the valuable the inputs. I will try to implement the same. Thanks. ---
**Author:** ![Manikanth\_Reddy](https://avatars.discourse-cdn.com/v4/letter/m/5fc32e/32.png) [@Manikanth\_Reddy](https://discuss.elastic.co/u/Manikanth_Reddy)\ **Post date:** [October 17, 2017, 6:30pm UTC](https://discuss.elastic.co/t/sharding-strategy/103573/11 "2017-10-17T18:30:29Z")
@Christian_Dahlqvist I have started implementing weekly indices. But I have quick doubt about 2 copies of each shard. As we have 3 data nodes, If I set my primary shard as 1 and replica as 1, how will I get 2 copies of each shard? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 17, 2017, 8:27pm UTC](https://discuss.elastic.co/t/sharding-strategy/103573/12 "2017-10-17T20:27:59Z")
If replica is set to 1, it means that you will have one replica in addition to the primary shards, resulting in two copies of the same shard. ---
**Author:** ![Manikanth\_Reddy](https://avatars.discourse-cdn.com/v4/letter/m/5fc32e/32.png) [@Manikanth\_Reddy](https://discuss.elastic.co/u/Manikanth_Reddy)\ **Post date:** [October 17, 2017, 8:32pm UTC](https://discuss.elastic.co/t/sharding-strategy/103573/13 "2017-10-17T20:32:31Z")
@Christian_Dahlqvist Thanks for the response. As I am going to have 1 primary shard and 1 replica, what if 2 data nodes were down(which are holding 1 primary and 1 replica respectively) out of 3 data nodes. How can i handle this failure? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 17, 2017, 8:35pm UTC](https://discuss.elastic.co/t/sharding-strategy/103573/14 "2017-10-17T20:35:03Z")
If 2 out of 3 nodes are down and they are all master eligible, the cluster should no longer be able to accept writes if configured correctly, as this could otherwise lead to data loss. If you need protection against 2 nodes going down you will need a cluster of at least 5 master-eligible nodes and set replicas to 2 so that you have 3 copies of each shard. ---
**Author:** ![Manikanth\_Reddy](https://avatars.discourse-cdn.com/v4/letter/m/5fc32e/32.png) [@Manikanth\_Reddy](https://discuss.elastic.co/u/Manikanth_Reddy)\ **Post date:** [October 17, 2017, 8:44pm UTC](https://discuss.elastic.co/t/sharding-strategy/103573/15 "2017-10-17T20:44:39Z")
Sure @Christian_Dahlqvist Thanks. We have 3 master nodes, 3 data nodes and 2 client nodes. As per our configuration, Data nodes are not eligible for becoming master. In order to protect against 2 nodes going down, I will try to setup replicas to 2. Thank for the help @Christian_Dahlqvist ---
**Author:** ![Manikanth](https://avatars.discourse-cdn.com/v4/letter/m/f0a364/32.png) [@Manikanth](https://discuss.elastic.co/u/Manikanth)\ **Post date:** [October 25, 2017, 1:38am UTC](https://discuss.elastic.co/t/sharding-strategy/103573/16 "2017-10-25T01:38:51Z")
@Christian_Dahlqvist Shards improve performance of the queries. But if i set primary shards to 1 and replica as 1 fro each weekly index, does this impact the performance of queries. ---
**Author:** ![Manikanth\_Reddy](https://avatars.discourse-cdn.com/v4/letter/m/5fc32e/32.png) [@Manikanth\_Reddy](https://discuss.elastic.co/u/Manikanth_Reddy)\ **Post date:** [October 25, 2017, 1:42am UTC](https://discuss.elastic.co/t/sharding-strategy/103573/17 "2017-10-25T01:42:50Z")
@Christian_Dahlqvist Shards improve performance of the queries. But if i set primary shards to 1 and replica as 1 fro each weekly index, does this impact the performance of queries. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 25, 2017, 9:56am UTC](https://discuss.elastic.co/t/sharding-strategy/103573/18 "2017-10-25T09:56:07Z")
You would need to test that, but more smaller shards does not necessarily mean faster queries. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 22, 2017, 9:56am UTC](https://discuss.elastic.co/t/sharding-strategy/103573/19 "2017-11-22T09:56:12Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.