# Strange rebalancing of shards **URL:** \ **Category:** Elasticsearch\ **Created:** [October 18, 2011, 12:46pm UTC](https://discuss.elastic.co/t/strange-rebalancing-of-shards/5624 "2011-10-18T12:46:37Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Steff](https://avatars.discourse-cdn.com/v4/letter/s/e36b37/32.png) [@Steff](https://discuss.elastic.co/u/Steff)\ **Post date:** [October 18, 2011, 12:46pm UTC](https://discuss.elastic.co/t/strange-rebalancing-of-shards/5624/1 "2011-10-18T12:46:37Z")
Hi We have made a simple test on rebalacing of shards. Start-state: One index with 3 shards (1 replica) Two nodes: Node1 running primary of shard1, primary of shard2 and replica of shard3 Node2 running primary of shard3, replica of shard1 and replica of shard2 Action: We start a new node (Node3) that joins the cluster (multicast) End-state (after rebalancing has finished) Three nodes: Node1 running replica of shard3 Node2 running replica of shard1 and replica of shard2 Node3 running primary of shard1, primary of shard2 and primary of shard3 Basically ALL primarys have been moved to the new node. We think that this is a very strange rebalancing of shards that ES decided to do. We would probably have excpeted something like this as end-state (a more "even" end-balance): Three nodes: Node1 running primary of shard1 and replica of shard3 Node2 running primary of shard3 and replica of shard2 Node3 running primary of shard2 and replica of shard1 Basically moving one primary and one replica over to Node3 Any comments on this "strange" behaviour? Is it intended, or do you have any explanation why it behaves like that? We repeated the test twice with the same result both times. Regards, Per Steffensen ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 18, 2011, 6:16pm UTC](https://discuss.elastic.co/t/strange-rebalancing-of-shards/5624/2 "2011-10-18T18:16:33Z")
How do you check where shards are allocated? Here is a simple gist that I used and I can see that with 3 nodes, and an index with 3 shards and 1 replica, shards are evenly distributed across the nodes (2 shards per node): [gist:1296201 · GitHub](https://gist.github.com/1296201). On Tue, Oct 18, 2011 at 2:46 PM, Steff [steff@designware.dk](mailto:steff@designware.dk) wrote: > Hi > > We have made a simple test on rebalacing of shards. > > Start-state: > One index with 3 shards (1 replica) > Two nodes: > Node1 running primary of shard1, primary of shard2 and replica of > shard3 > Node2 running primary of shard3, replica of shard1 and replica of > shard2 > > Action: > We start a new node (Node3) that joins the cluster (multicast) > > End-state (after rebalancing has finished) > Three nodes: > Node1 running replica of shard3 > Node2 running replica of shard1 and replica of shard2 > Node3 running primary of shard1, primary of shard2 and primary of > shard3 > Basically ALL primarys have been moved to the new node. > > We think that this is a very strange rebalancing of shards that ES > decided to do. We would probably have excpeted something like this as > end-state (a more "even" end-balance): > Three nodes: > Node1 running primary of shard1 and replica of shard3 > Node2 running primary of shard3 and replica of shard2 > Node3 running primary of shard2 and replica of shard1 > Basically moving one primary and one replica over to Node3 > > Any comments on this "strange" behaviour? Is it intended, or do you > have any explanation why it behaves like that? > > We repeated the test twice with the same result both times. > > Regards, Per Steffensen ---
**Author:** ![Steff](https://avatars.discourse-cdn.com/v4/letter/s/e36b37/32.png) [@Steff](https://discuss.elastic.co/u/Steff)\ **Post date:** [October 24, 2011, 5:36pm UTC](https://discuss.elastic.co/t/strange-rebalancing-of-shards/5624/3 "2011-10-24T17:36:40Z")
On Oct 18, 8:16 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > How do you check where shards are allocated? I use elasticsearch-head ([http://mobz.github.com/elasticsearch-head/](http://mobz.github.com/elasticsearch-head/)) > Here is a simple gist that I > used and I can see that with 3 nodes, and an index with 3 shards and 1 > replica, shards are evenly distributed across the nodes (2 shards per > node):[gist:1296201 · GitHub](https://gist.github.com/1296201). Hmm, it was not what we saw. Just leave this issue for now, until I get to repeat the test to see if it is a consistent problem and maybe find out what causes it. ---
**Author:** ![Steff](https://avatars.discourse-cdn.com/v4/letter/s/e36b37/32.png) [@Steff](https://discuss.elastic.co/u/Steff)\ **Post date:** [October 26, 2011, 7:15am UTC](https://discuss.elastic.co/t/strange-rebalancing-of-shards/5624/4 "2011-10-26T07:15:57Z")
gateway.expected\_nodes might have been set to 2 all the way through the test. Could that be the cause of the "problem"? Regards, Per Steffensen ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 26, 2011, 5:01pm UTC](https://discuss.elastic.co/t/strange-rebalancing-of-shards/5624/5 "2011-10-26T17:01:06Z")
No, it shouldn't, it just control when the balancing / recovery will start, not how it will happen... . On Wed, Oct 26, 2011 at 9:15 AM, Steff [steff@designware.dk](mailto:steff@designware.dk) wrote: > gateway.expected\_nodes might have been set to 2 all the way through > the test. Could that be the cause of the "problem"? > > Regards, Per Steffensen ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:50am UTC](https://discuss.elastic.co/t/strange-rebalancing-of-shards/5624/6 "2017-07-06T03:50:50Z")