# Managing shard distribution **URL:** \ **Category:** Elasticsearch\ **Created:** [April 1, 2013, 5:03pm UTC](https://discuss.elastic.co/t/managing-shard-distribution/11402 "2013-04-01T17:03:36Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Dave\_Rawks\_2](https://avatars.discourse-cdn.com/v4/letter/d/b5ac83/32.png) [@Dave\_Rawks\_2](https://discuss.elastic.co/u/Dave_Rawks_2)\ **Post date:** [April 1, 2013, 5:03pm UTC](https://discuss.elastic.co/t/managing-shard-distribution/11402/1 "2013-04-01T17:03:36Z")
Is there any way to configure elasticsearch such that all the given shards and replicas for a give index will be evenly and equally distributed across all nodes? I've got a 6 node cluster and all ym indices are configured for 6 shards and 1 replica; it would seem to me that they should distribute such that each node has a single primary and a single replica shard for each index. However in practice the distribution scheme seems to be ignorant of indices resulting in most indices being spread across at most 2 or 3 of the nodes. My indices hold log data and the general query usage hits 1, 7, 30, or 90 indices. If the shards/replicas are distributed with some index awareness it seems like I'll get better distribution of load on queries regardless of the number of indices being queried. Setting the max shards per node option sort of works, the shard routing will sometimes make poor decisions resulting in a single shard unable to find a suitable node AND there is no flexibility to the option allowing for replication to ensure the extra replicas in the case that one node disappears. Any help would be muchly appreciated. -Dave -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [April 1, 2013, 10:12pm UTC](https://discuss.elastic.co/t/managing-shard-distribution/11402/2 "2013-04-01T22:12:57Z")
Using 0.20.x or earlier, your only option is to set total\_shards\_per\_node. I have some custom code in my app that re-adjusts this based on data node count and if things do get stuck in yellow like you mention, toggles this number up 1 and then down for that specific index. If you're on 0.90.x or later, ES has this built in: > > > \- Weights are calculated per index and incorporate index level, global and prima…ry related parameters > \- Balance operations are executed based on a win maximation strategy that tries to relocate shards first that offer the biggest gain towards the weight functions optimum > \- The WeightFunction allows settings to prefer index based balance over global balance and vice versa > \- Balance operations can be throttled by raising a threshold resulting in less agressive balance operations > \- WeightFunction shipps with defaults to achive evenly distributed indexes while maintaining a global balance > > This closes issue #2555 Best Regards, Paul On Monday, April 1, 2013 11:03:36 AM UTC-6, Dave Rawks wrote: > Is there any way to configure elasticsearch such that all the given shards > and replicas for a give index will be evenly and equally distributed across > all nodes? I've got a 6 node cluster and all ym indices are configured for > 6 shards and 1 replica; it would seem to me that they should distribute > such that each node has a single primary and a single replica shard for > each index. However in practice the distribution scheme seems to be > ignorant of indices resulting in most indices being spread across at most 2 > or 3 of the nodes. My indices hold log data and the general query usage > hits 1, 7, 30, or 90 indices. If the shards/replicas are distributed with > some index awareness it seems like I'll get better distribution of load on > queries regardless of the number of indices being queried. > > Setting the max shards per node option sort of works, the shard routing > will sometimes make poor decisions resulting in a single shard unable to > find a suitable node AND there is no flexibility to the option allowing for > replication to ensure the extra replicas in the case that one node > disappears. > > Any help would be muchly appreciated. > > -Dave -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Mohammady\_Mahdy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohammady_mahdy/32/2128_2.png) [@Mohammady\_Mahdy](https://discuss.elastic.co/u/Mohammady_Mahdy)\ **Post date:** [April 2, 2013, 7:35am UTC](https://discuss.elastic.co/t/managing-shard-distribution/11402/3 "2013-04-02T07:35:11Z")
@ppearcy is this something you could share? do you have an idea of what causes things to get stuck in red? On Tuesday, April 2, 2013 2:12:57 AM UTC+4, ppearcy wrote: > Using 0.20.x or earlier, your only option is to set total\_shards\_per\_node. > I have some custom code in my app that re-adjusts this based on data node > count and if things do get stuck in yellow like you mention, toggles this > number up 1 and then down for that specific index. > > If you're on 0.90.x or later, ES has this built in: > [#2555 Added BalancedShardsAllocator that balances shards based on a weight function. by s1monw · Pull Request #2556 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/pull/2556) > > Best Regards, > Paul > > On Monday, April 1, 2013 11:03:36 AM UTC-6, Dave Rawks wrote: > > > Is there any way to configure elasticsearch such that all the given > > shards and replicas for a give index will be evenly and equally distributed > > across all nodes? I've got a 6 node cluster and all ym indices are > > configured for 6 shards and 1 replica; it would seem to me that they should > > distribute such that each node has a single primary and a single replica > > shard for each index. However in practice the distribution scheme seems to > > be ignorant of indices resulting in most indices being spread across at > > most 2 or 3 of the nodes. My indices hold log data and the general query > > usage hits 1, 7, 30, or 90 indices. If the shards/replicas are distributed > > with some index awareness it seems like I'll get better distribution of > > load on queries regardless of the number of indices being queried. > > > > Setting the max shards per node option sort of works, the shard routing > > will sometimes make poor decisions resulting in a single shard unable to > > find a suitable node AND there is no flexibility to the option allowing for > > replication to ensure the extra replicas in the case that one node > > disappears. > > > > Any help would be muchly appreciated. > > > > -Dave -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Mohammady\_Mahdy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohammady_mahdy/32/2128_2.png) [@Mohammady\_Mahdy](https://discuss.elastic.co/u/Mohammady_Mahdy)\ **Post date:** [April 2, 2013, 7:36am UTC](https://discuss.elastic.co/t/managing-shard-distribution/11402/4 "2013-04-02T07:36:02Z")
Hi Paul, Thanks for your response. @ppearcy is this code something you could share? do you have an idea of what causes things to get stuck in yellow? On Tuesday, April 2, 2013 2:12:57 AM UTC+4, ppearcy wrote: > Using 0.20.x or earlier, your only option is to set total\_shards\_per\_node. > I have some custom code in my app that re-adjusts this based on data node > count and if things do get stuck in yellow like you mention, toggles this > number up 1 and then down for that specific index. > > If you're on 0.90.x or later, ES has this built in: > [#2555 Added BalancedShardsAllocator that balances shards based on a weight function. by s1monw · Pull Request #2556 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/pull/2556) > > Best Regards, > Paul > > On Monday, April 1, 2013 11:03:36 AM UTC-6, Dave Rawks wrote: > > > Is there any way to configure elasticsearch such that all the given > > shards and replicas for a give index will be evenly and equally distributed > > across all nodes? I've got a 6 node cluster and all ym indices are > > configured for 6 shards and 1 replica; it would seem to me that they should > > distribute such that each node has a single primary and a single replica > > shard for each index. However in practice the distribution scheme seems to > > be ignorant of indices resulting in most indices being spread across at > > most 2 or 3 of the nodes. My indices hold log data and the general query > > usage hits 1, 7, 30, or 90 indices. If the shards/replicas are distributed > > with some index awareness it seems like I'll get better distribution of > > load on queries regardless of the number of indices being queried. > > > > Setting the max shards per node option sort of works, the shard routing > > will sometimes make poor decisions resulting in a single shard unable to > > find a suitable node AND there is no flexibility to the option allowing for > > replication to ensure the extra replicas in the case that one node > > disappears. > > > > Any help would be muchly appreciated. > > > > -Dave -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [April 2, 2013, 5:26pm UTC](https://discuss.elastic.co/t/managing-shard-distribution/11402/5 "2013-04-02T17:26:56Z")
Sure... You should be able to drop this code into a background thread process that runs every so often. Will need to make at least a couple of tweaks: - You will not have an AMS class (that is for some internal monitoring system we have) - You won't have an ESIndexer class that holds the elasticsearch client Use at your own risk, no guarantees, yadayadayada 🙂 > Best Regards, Paul On Tuesday, April 2, 2013 1:36:02 AM UTC-6, Mo wrote: > Hi Paul, > > Thanks for your response. > > @ppearcy is this code something you could share? do you have an idea of > what causes things to get stuck in yellow? > > On Tuesday, April 2, 2013 2:12:57 AM UTC+4, ppearcy wrote: > > > Using 0.20.x or earlier, your only option is to set > > total\_shards\_per\_node. I have some custom code in my app that re-adjusts > > this based on data node count and if things do get stuck in yellow like you > > mention, toggles this number up 1 and then down for that specific index. > > > > If you're on 0.90.x or later, ES has this built in: > > [#2555 Added BalancedShardsAllocator that balances shards based on a weight function. by s1monw · Pull Request #2556 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/pull/2556) > > > > Best Regards, > > Paul > > > > On Monday, April 1, 2013 11:03:36 AM UTC-6, Dave Rawks wrote: > > > > > Is there any way to configure elasticsearch such that all the given > > > shards and replicas for a give index will be evenly and equally distributed > > > across all nodes? I've got a 6 node cluster and all ym indices are > > > configured for 6 shards and 1 replica; it would seem to me that they should > > > distribute such that each node has a single primary and a single replica > > > shard for each index. However in practice the distribution scheme seems to > > > be ignorant of indices resulting in most indices being spread across at > > > most 2 or 3 of the nodes. My indices hold log data and the general query > > > usage hits 1, 7, 30, or 90 indices. If the shards/replicas are distributed > > > with some index awareness it seems like I'll get better distribution of > > > load on queries regardless of the number of indices being queried. > > > > > > Setting the max shards per node option sort of works, the shard routing > > > will sometimes make poor decisions resulting in a single shard unable to > > > find a suitable node AND there is no flexibility to the option allowing for > > > replication to ensure the extra replicas in the case that one node > > > disappears. > > > > > > Any help would be muchly appreciated. > > > > > > -Dave -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Jilles\_van\_Gurp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jilles_van_gurp/32/879_2.png) [@Jilles\_van\_Gurp](https://discuss.elastic.co/u/Jilles_van_Gurp)\ **Post date:** [April 3, 2013, 7:05am UTC](https://discuss.elastic.co/t/managing-shard-distribution/11402/6 "2013-04-03T07:05:34Z")
The upcoming 0.90 release has a vastly improved way of allocating shards. So, if you are considering custom solutions, you might want to upgrade to the release candidate and see if that solves your problem. Jilles On Monday, April 1, 2013 7:03:36 PM UTC+2, Dave Rawks wrote: > Is there any way to configure elasticsearch such that all the given shards > and replicas for a give index will be evenly and equally distributed across > all nodes? I've got a 6 node cluster and all ym indices are configured for > 6 shards and 1 replica; it would seem to me that they should distribute > such that each node has a single primary and a single replica shard for > each index. However in practice the distribution scheme seems to be > ignorant of indices resulting in most indices being spread across at most 2 > or 3 of the nodes. My indices hold log data and the general query usage > hits 1, 7, 30, or 90 indices. If the shards/replicas are distributed with > some index awareness it seems like I'll get better distribution of load on > queries regardless of the number of indices being queried. > > Setting the max shards per node option sort of works, the shard routing > will sometimes make poor decisions resulting in a single shard unable to > find a suitable node AND there is no flexibility to the option allowing for > replication to ensure the extra replicas in the case that one node > disappears. > > Any help would be muchly appreciated. > > -Dave -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Mohammady\_Mahdy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohammady_mahdy/32/2128_2.png) [@Mohammady\_Mahdy](https://discuss.elastic.co/u/Mohammady_Mahdy)\ **Post date:** [April 3, 2013, 7:40am UTC](https://discuss.elastic.co/t/managing-shard-distribution/11402/7 "2013-04-03T07:40:16Z")
many thanks 🙂 On Tuesday, April 2, 2013 9:26:56 PM UTC+4, ppearcy wrote: > Sure... You should be able to drop this code into a background thread > process that runs every so often. Will need to make at least a couple of > tweaks: > > - You will not have an AMS class (that is for some internal monitoring > system we have) > - You won't have an ESIndexer class that holds the elasticsearch client > > Use at your own risk, no guarantees, yadayadayada 🙂 > > [Code to dynamically set the number of shards per node for each elasticsearch index. · GitHub](https://gist.github.com/ppearcy/5294200) > > Best Regards, > Paul > > On Tuesday, April 2, 2013 1:36:02 AM UTC-6, Mo wrote: > > > Hi Paul, > > > > Thanks for your response. > > > > @ppearcy is this code something you could share? do you have an idea of > > what causes things to get stuck in yellow? > > > > On Tuesday, April 2, 2013 2:12:57 AM UTC+4, ppearcy wrote: > > > > > Using 0.20.x or earlier, your only option is to set > > > total\_shards\_per\_node. I have some custom code in my app that re-adjusts > > > this based on data node count and if things do get stuck in yellow like you > > > mention, toggles this number up 1 and then down for that specific index. > > > > > > If you're on 0.90.x or later, ES has this built in: > > > [#2555 Added BalancedShardsAllocator that balances shards based on a weight function. by s1monw · Pull Request #2556 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/pull/2556) > > > > > > Best Regards, > > > Paul > > > > > > On Monday, April 1, 2013 11:03:36 AM UTC-6, Dave Rawks wrote: > > > > > > > Is there any way to configure elasticsearch such that all the given > > > > shards and replicas for a give index will be evenly and equally distributed > > > > across all nodes? I've got a 6 node cluster and all ym indices are > > > > configured for 6 shards and 1 replica; it would seem to me that they should > > > > distribute such that each node has a single primary and a single replica > > > > shard for each index. However in practice the distribution scheme seems to > > > > be ignorant of indices resulting in most indices being spread across at > > > > most 2 or 3 of the nodes. My indices hold log data and the general query > > > > usage hits 1, 7, 30, or 90 indices. If the shards/replicas are distributed > > > > with some index awareness it seems like I'll get better distribution of > > > > load on queries regardless of the number of indices being queried. > > > > > > > > Setting the max shards per node option sort of works, the shard routing > > > > will sometimes make poor decisions resulting in a single shard unable to > > > > find a suitable node AND there is no flexibility to the option allowing for > > > > replication to ensure the extra replicas in the case that one node > > > > disappears. > > > > > > > > Any help would be muchly appreciated. > > > > > > > > -Dave -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:43am UTC](https://discuss.elastic.co/t/managing-shard-distribution/11402/8 "2017-07-06T02:43:06Z")