# Fair distribution of shards per node per index **URL:** \ **Category:** Elasticsearch\ **Created:** [October 31, 2011, 10:19am UTC](https://discuss.elastic.co/t/fair-distribution-of-shards-per-node-per-index/5724 "2011-10-31T10:19:55Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Steff](https://avatars.discourse-cdn.com/v4/letter/s/e36b37/32.png) [@Steff](https://discuss.elastic.co/u/Steff)\ **Post date:** [October 31, 2011, 10:19am UTC](https://discuss.elastic.co/t/fair-distribution-of-shards-per-node-per-index/5724/1 "2011-10-31T10:19:55Z")
I believe the current shards-distribution algoritm in ES aims at having number\_of\_shards\_all\_in\_all/number\_of\_nodes shards running on each node. We are using one index for each month, so that all data that "belongs" to a certain month is indexed into a particular index representing that month. 99.99% percent of all data that is index into ES during a concrete month "belongs" to that same month, so 99.99% of the indexing going on during a concrete day works only against one single index. All the other indices we keep around for "history" purposes so that you can query back in time in historic data - we are querying our data very seldom. Because most of the indexing that goes on during a day goes on against one single index, and because almost all of the work we do against ES is about indexing new documents, we would like that the shards of that particular index is fairly distributed among nodes. That does not seem to be true. Basically we would like fair distribution af number\_of\_shards/number\_of\_nodes PER INDEX. Does ES support that? Or maybe there are a way to manually ask ES to relocate shards (maybe only supporting swapping indices to keep the number\_of\_shards\_all\_in\_all/number\_of\_nodes invariant) Regards, Per Steffensen ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 31, 2011, 5:58pm UTC](https://discuss.elastic.co/t/fair-distribution-of-shards-per-node-per-index/5724/2 "2011-10-31T17:58:24Z")
Heya, Yea, this is an area where elasticsearch should have better behavior. There isn't an option ot have "per index" distribution, or weighted distribution. There isn't a good answer now, except maybe for the ability to control where an index can be placed (which nodes). On Mon, Oct 31, 2011 at 12:19 PM, Steff [steff@designware.dk](mailto:steff@designware.dk) wrote: > I believe the current shards-distribution algoritm in ES aims at > having number\_of\_shards\_all\_in\_all/number\_of\_nodes shards running on > each node. We are using one index for each month, so that all data > that "belongs" to a certain month is indexed into a particular index > representing that month. 99.99% percent of all data that is index into > ES during a concrete month "belongs" to that same month, so 99.99% of > the indexing going on during a concrete day works only against one > single index. All the other indices we keep around for "history" > purposes so that you can query back in time in historic data - we are > querying our data very seldom. > > Because most of the indexing that goes on during a day goes on against > one single index, and because almost all of the work we do against ES > is about indexing new documents, we would like that the shards of that > particular index is fairly distributed among nodes. That does not seem > to be true. Basically we would like fair distribution af > number\_of\_shards/number\_of\_nodes PER INDEX. Does ES support that? Or > maybe there are a way to manually ask ES to relocate shards (maybe > only supporting swapping indices to keep the > number\_of\_shards\_all\_in\_all/number\_of\_nodes invariant) > > Regards, Per Steffensen ---
**Author:** ![Mohit\_Anchlia](https://avatars.discourse-cdn.com/v4/letter/m/848f3c/32.png) [@Mohit\_Anchlia](https://discuss.elastic.co/u/Mohit_Anchlia)\ **Post date:** [October 31, 2011, 7:45pm UTC](https://discuss.elastic.co/t/fair-distribution-of-shards-per-node-per-index/5724/3 "2011-10-31T19:45:00Z")
On Mon, Oct 31, 2011 at 10:58 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > Heya, > Yea, this is an area where elasticsearch should have better behavior. > There isn't an option ot have "per index" distribution, or weighted > distribution. There isn't a good answer now, except maybe for the ability to > control where an index can be placed (which nodes). I thought sharding is done based on the document id hash in a given index. If that is true then creating new documents would shard the data? > On Mon, Oct 31, 2011 at 12:19 PM, Steff [steff@designware.dk](mailto:steff@designware.dk) wrote: > > > I believe the current shards-distribution algoritm in ES aims at > > having number\_of\_shards\_all\_in\_all/number\_of\_nodes shards running on > > each node. We are using one index for each month, so that all data > > that "belongs" to a certain month is indexed into a particular index > > representing that month. 99.99% percent of all data that is index into > > ES during a concrete month "belongs" to that same month, so 99.99% of > > the indexing going on during a concrete day works only against one > > single index. All the other indices we keep around for "history" > > purposes so that you can query back in time in historic data - we are > > querying our data very seldom. > > > > Because most of the indexing that goes on during a day goes on against > > one single index, and because almost all of the work we do against ES > > is about indexing new documents, we would like that the shards of that > > particular index is fairly distributed among nodes. That does not seem > > to be true. Basically we would like fair distribution af > > number\_of\_shards/number\_of\_nodes PER INDEX. Does ES support that? Or > > maybe there are a way to manually ask ES to relocate shards (maybe > > only supporting swapping indices to keep the > > number\_of\_shards\_all\_in\_all/number\_of\_nodes invariant) > > > > Regards, Per Steffensen ---
**Author:** ![Steff](https://avatars.discourse-cdn.com/v4/letter/s/e36b37/32.png) [@Steff](https://discuss.elastic.co/u/Steff)\ **Post date:** [November 1, 2011, 7:30am UTC](https://discuss.elastic.co/t/fair-distribution-of-shards-per-node-per-index/5724/4 "2011-11-01T07:30:05Z")
On 31 Okt., 18:58, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > Heya, > > Yea, this is an area where elasticsearch should have better behavior. > There isn't an option ot have "per index" distribution, or weighted > distribution. There isn't a good answer now, except maybe for the ability > to control where an index can be placed (which nodes). Ok, thanks. Well it is open source, so maybe we will make something ourselves. I didnt know about the abilitiy to control where an index can be placed - can you give short explanation about how to control, or maybe a pointer to a web-page explaining? Thanks ---
**Author:** ![Steff](https://avatars.discourse-cdn.com/v4/letter/s/e36b37/32.png) [@Steff](https://discuss.elastic.co/u/Steff)\ **Post date:** [November 1, 2011, 7:31am UTC](https://discuss.elastic.co/t/fair-distribution-of-shards-per-node-per-index/5724/5 "2011-11-01T07:31:26Z")
> I thought sharding is done based on the document id hash in a given > index. If that is true then creating new documents > would shard the data? Yes documents are distributed among shards based on document id (by default at least), but my problem is not that documents are not distributed well among shards. The problem is that shards are not distributed well among nodes. Regards, Per Steffensen ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [November 1, 2011, 5:46pm UTC](https://discuss.elastic.co/t/fair-distribution-of-shards-per-node-per-index/5724/6 "2011-11-01T17:46:14Z")
On Mon, Oct 31, 2011 at 9:45 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)wrote: > On Mon, Oct 31, 2011 at 10:58 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > Heya, > > Yea, this is an area where elasticsearch should have better behavior. > > There isn't an option ot have "per index" distribution, or weighted > > distribution. There isn't a good answer now, except maybe for the > > ability to > > control where an index can be placed (which nodes). > > I thought sharding is done based on the document id hash in a given > index. If that is true then creating new documents > would shard the data? Yes, thats what happens. But, then there is a question of how shards are allocated among nodes. The way elasticsearch does that now is by aiming to get an even number of shards per node. Thats not always the desirable distribution (i.e. size based one, for example). > > On Mon, Oct 31, 2011 at 12:19 PM, Steff [steff@designware.dk](mailto:steff@designware.dk) wrote: > > > > > I believe the current shards-distribution algoritm in ES aims at > > > having number\_of\_shards\_all\_in\_all/number\_of\_nodes shards running on > > > each node. We are using one index for each month, so that all data > > > that "belongs" to a certain month is indexed into a particular index > > > representing that month. 99.99% percent of all data that is index into > > > ES during a concrete month "belongs" to that same month, so 99.99% of > > > the indexing going on during a concrete day works only against one > > > single index. All the other indices we keep around for "history" > > > purposes so that you can query back in time in historic data - we are > > > querying our data very seldom. > > > > > > Because most of the indexing that goes on during a day goes on against > > > one single index, and because almost all of the work we do against ES > > > is about indexing new documents, we would like that the shards of that > > > particular index is fairly distributed among nodes. That does not seem > > > to be true. Basically we would like fair distribution af > > > number\_of\_shards/number\_of\_nodes PER INDEX. Does ES support that? Or > > > maybe there are a way to manually ask ES to relocate shards (maybe > > > only supporting swapping indices to keep the > > > number\_of\_shards\_all\_in\_all/number\_of\_nodes invariant) > > > > > > Regards, Per Steffensen ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [November 1, 2011, 5:47pm UTC](https://discuss.elastic.co/t/fair-distribution-of-shards-per-node-per-index/5724/7 "2011-11-01T17:47:43Z")
On Tue, Nov 1, 2011 at 9:30 AM, Steff [steff@designware.dk](mailto:steff@designware.dk) wrote: > On 31 Okt., 18:58, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > Heya, > > > > Yea, this is an area where elasticsearch should have better behavior. > > There isn't an option ot have "per index" distribution, or weighted > > distribution. There isn't a good answer now, except maybe for the ability > > to control where an index can be placed (which nodes). > > Ok, thanks. Well it is open source, so maybe we will make something > ourselves. I didnt know about the abilitiy to control where an index > can be placed - can you give short explanation about how to control, > or maybe a pointer to a web-page explaining? Sure (note, its a new feature in 0.18): [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/cluster.html) (see Shard Allocation Filtering). > Thanks ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:50am UTC](https://discuss.elastic.co/t/fair-distribution-of-shards-per-node-per-index/5724/8 "2017-07-06T03:50:14Z")