# Unbalanced cluster - one node running out of space

**URL:** https://discuss.elastic.co/t/unbalanced-cluster-one-node-running-out-of-space/53890
**Category:** Elasticsearch
**Created:** [June 24, 2016, 1:28pm UTC](https://discuss.elastic.co/t/unbalanced-cluster-one-node-running-out-of-space/53890 "2016-06-24T13:28:47Z")
**Posts on this page:** 8
**Page:** 1

<div class="post-metadata">

### Author: ![elssar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/elssar/32/9216_2.png) [@elssar](https://discuss.elastic.co/u/elssar)
#### Post date: [June 24, 2016, 1:28pm UTC](https://discuss.elastic.co/t/unbalanced-cluster-one-node-running-out-of-space/53890/1 "2016-06-24T13:28:47Z")

</div>

I have a cluster with 1 dedicated master node, 1 client node, and 5 data nodes..

While running a particularly heavy aggregation, the cluster turned yellow and a few replica shards got stuck in `Initializing` state. After the aggregation was ran again, a lot of replica shards (~1/6th ) became unassigned.

Now I see that one of the data nodes has been assigned a bulk of the primary shards and it is running out of disk space fast.

All data nodes have 500gb disks, and the other 4 have between 100 and 150 GB of disk space free. The 5th one has less than 20.

How can I remedy this? Will moving shards away from this node help?

EDIT: I'm using Elasticsearch version 1.7.1

---

<div class="post-metadata">

### Author: ![Josh\_J\_Luo](https://avatars.discourse-cdn.com/v4/letter/j/dbc845/32.png) [@Josh\_J\_Luo](https://discuss.elastic.co/u/Josh_J_Luo)
#### Post date: [June 24, 2016, 2:13pm UTC](https://discuss.elastic.co/t/unbalanced-cluster-one-node-running-out-of-space/53890/2 "2016-06-24T14:13:22Z")

</div>

I would suggest you to first check your \_routing field

[https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-routing-field.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-routing-field.html)

There is a chance that your ids (by default \_routing field uses ids) are not well distributed, causing the routing to crowd certain nodes.

---

<div class="post-metadata">

### Author: ![elssar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/elssar/32/9216_2.png) [@elssar](https://discuss.elastic.co/u/elssar)
#### Post date: [June 24, 2016, 2:27pm UTC](https://discuss.elastic.co/t/unbalanced-cluster-one-node-running-out-of-space/53890/3 "2016-06-24T14:27:51Z")

</div>

Elasticsearch auto generates ids, so I doubt thats the case

---

<div class="post-metadata">

### Author: ![Josh\_J\_Luo](https://avatars.discourse-cdn.com/v4/letter/j/dbc845/32.png) [@Josh\_J\_Luo](https://discuss.elastic.co/u/Josh_J_Luo)
#### Post date: [June 24, 2016, 2:29pm UTC](https://discuss.elastic.co/t/unbalanced-cluster-one-node-running-out-of-space/53890/4 "2016-06-24T14:29:12Z")

</div>

Do you have parent-child relationship in your index?

---

<div class="post-metadata">

### Author: ![elssar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/elssar/32/9216_2.png) [@elssar](https://discuss.elastic.co/u/elssar)
#### Post date: [June 24, 2016, 2:31pm UTC](https://discuss.elastic.co/t/unbalanced-cluster-one-node-running-out-of-space/53890/5 "2016-06-24T14:31:24Z")

</div>

No. Its a standard logstash populated index

---

<div class="post-metadata">

### Author: ![Josh\_J\_Luo](https://avatars.discourse-cdn.com/v4/letter/j/dbc845/32.png) [@Josh\_J\_Luo](https://discuss.elastic.co/u/Josh_J_Luo)
#### Post date: [June 24, 2016, 2:59pm UTC](https://discuss.elastic.co/t/unbalanced-cluster-one-node-running-out-of-space/53890/6 "2016-06-24T14:59:21Z")

</div>

If so, probably you hit a bug (or hit by one). See if this helps [https://github.com/elastic/elasticsearch/pull/14494](https://github.com/elastic/elasticsearch/pull/14494)

---

<div class="post-metadata">

### Author: ![elssar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/elssar/32/9216_2.png) [@elssar](https://discuss.elastic.co/u/elssar)
#### Post date: [June 25, 2016, 5:31am UTC](https://discuss.elastic.co/t/unbalanced-cluster-one-node-running-out-of-space/53890/7 "2016-06-25T05:31:04Z")

</div>

Thanks @Josh_J_Luo, I'll take a look.

The cluster recovered on its own though. The problematic node went from almost running out of disk space (\<1gb) to having over a 100gb free.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 10:40pm UTC](https://discuss.elastic.co/t/unbalanced-cluster-one-node-running-out-of-space/53890/8 "2017-07-05T22:40:36Z")

</div>


