# All shards being allocated on the same node

**URL:** <https://discuss.elastic.co/t/all-shards-being-allocated-on-the-same-node/52880>\
**Category:** Elasticsearch\
**Created:** [June 15, 2016, 2:59pm UTC](https://discuss.elastic.co/t/all-shards-being-allocated-on-the-same-node/52880 "2016-06-15T14:59:12Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![rkrombho](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkrombho/32/3574_2.png) [@rkrombho](https://discuss.elastic.co/u/rkrombho)\
**Post date:** [June 15, 2016, 2:59pm UTC](https://discuss.elastic.co/t/all-shards-being-allocated-on-the-same-node/52880/1 "2016-06-15T14:59:12Z")

</div>

Hi All,

after quite some time running a cluster with 444 indices I suddenly experience a problem while creating new indices.  
All Shards get allocated on the same node when I create new indices (replication = 0).

It's an 8 node cluster and I use `index.routing.allocation.require.storage_type` to only assign new indices to 4 of them.  
The `/_nodes` api shows that all 4 nodes have this attribute `storage_type` set correctly (and in the past it worked fine).

I'm below the disk watermarks and I can not find a reason why Elasticsearch would allocate all shards onto the same node.

I'm on ES 2.2

Someone got an idea on how I can debug this wired shard allocation problem?

Cheers  
Robert

---

<div class="post-metadata">

**Author:** ![ywelsch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ywelsch/32/7751_2.png) [@ywelsch](https://discuss.elastic.co/u/ywelsch)\
**Post date:** [June 15, 2016, 3:06pm UTC](https://discuss.elastic.co/t/all-shards-being-allocated-on-the-same-node/52880/2 "2016-06-15T15:06:58Z")

</div>

Is it possible that the node that receives the new shards has less shards than the other ones with same storage\_type? Can you check how many shards each of the 4 nodes has?

---

<div class="post-metadata">

**Author:** ![rkrombho](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkrombho/32/3574_2.png) [@rkrombho](https://discuss.elastic.co/u/rkrombho)\
**Post date:** [June 15, 2016, 3:11pm UTC](https://discuss.elastic.co/t/all-shards-being-allocated-on-the-same-node/52880/3 "2016-06-15T15:11:19Z")

</div>

yes you're right. The other 3 have 520 shards but this one (who gets all new ones) only got 344.  
Actually I have no idea why this is not overall balanced.

Also I thought that shard allocation balancing works on per-index base.

Can you recommend any solution to this state?  
Should I manually reroute a lot of shards?

---

<div class="post-metadata">

**Author:** ![ywelsch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ywelsch/32/7751_2.png) [@ywelsch](https://discuss.elastic.co/u/ywelsch)\
**Post date:** [June 15, 2016, 3:17pm UTC](https://discuss.elastic.co/t/all-shards-being-allocated-on-the-same-node/52880/4 "2016-06-15T15:17:34Z")

</div>

shard balancing works on the basis of 2 factors. One is trying to spread shards of same index, the other is trying to assign the same number of shards to each node. In this case, the node balance is so skewed that the index spread factor does not play a role. One possibility is to adapt the two factors (see [https://www.elastic.co/guide/en/elasticsearch/reference/2.3/shards-allocation.html#\_shard\_balancing\_heuristics](https://www.elastic.co/guide/en/elasticsearch/reference/2.3/shards-allocation.html#_shard_balancing_heuristics) ). The better solution is to investigate why 3 of the nodes have more shards than the other one. Are there any active relocations going on in your cluster? (Maybe node 4 was down and the cluster is still trying to balance shards to the node).

---

<div class="post-metadata">

**Author:** ![rkrombho](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkrombho/32/3574_2.png) [@rkrombho](https://discuss.elastic.co/u/rkrombho)\
**Post date:** [June 15, 2016, 3:30pm UTC](https://discuss.elastic.co/t/all-shards-being-allocated-on-the-same-node/52880/5 "2016-06-15T15:30:04Z")

</div>

Thanks for the explanation.  
Ah, I just noticed that the node with less shards actually is close to the low disk watermark.  
This is probably the reason.  
Can I expect ES to re-balance automatically when I free up disk space on this node?

---

<div class="post-metadata">

**Author:** ![ywelsch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ywelsch/32/7751_2.png) [@ywelsch](https://discuss.elastic.co/u/ywelsch)\
**Post date:** [June 15, 2016, 3:33pm UTC](https://discuss.elastic.co/t/all-shards-being-allocated-on-the-same-node/52880/6 "2016-06-15T15:33:34Z")

</div>

yes. Shards should start relocating to the node pretty quickly after that (disk space usage is polled every 30 seconds).

---

<div class="post-metadata">

**Author:** ![rkrombho](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkrombho/32/3574_2.png) [@rkrombho](https://discuss.elastic.co/u/rkrombho)\
**Post date:** [June 15, 2016, 4:15pm UTC](https://discuss.elastic.co/t/all-shards-being-allocated-on-the-same-node/52880/7 "2016-06-15T16:15:56Z")

</div>

Great. Thank you!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:43pm UTC](https://discuss.elastic.co/t/all-shards-being-allocated-on-the-same-node/52880/8 "2017-07-05T22:43:34Z")

</div>


