# Unbalanced disk usage with ES 6.1.3

**URL:** https://discuss.elastic.co/t/unbalanced-disk-usage-with-es-6-1-3/125193
**Category:** Elasticsearch
**Created:** [March 22, 2018, 1:10pm UTC](https://discuss.elastic.co/t/unbalanced-disk-usage-with-es-6-1-3/125193 "2018-03-22T13:10:50Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![CB193](https://avatars.discourse-cdn.com/v4/letter/c/c77e96/32.png) [@CB193](https://discuss.elastic.co/u/CB193)
#### Post date: [March 22, 2018, 1:10pm UTC](https://discuss.elastic.co/t/unbalanced-disk-usage-with-es-6-1-3/125193/1 "2018-03-22T13:10:50Z")

</div>

I have a 7 node cluster which was recently upgraded from 5.5 to 5.6 and then to 6.1.3. For some reason 3 (a,b,c) of the nodes have hundreds of gb of disk space being taken up and I can't find out why. This is causing the shards to not be balanced correctly.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/e/5e0c2992d8d98f0ddabd22343d97bb408a954e7d.png)

All the nodes are running Windows 2012 r2 and the indexes are on a dedicated drive.

Does anyone know of a way to find out what is using the space?

My thought was to pull one of the A B C nodes out of the cluster, let the cluster rebalance, delete all the index data from that node and then add it back into the cluster. My hope is that data would then be reallocated back the node in the proper amt. Does this make sense? Any other ideas?

This question is similar to

> [@Unbalanced disk usage with ES 2.4.x](https://discuss.elastic.co/t/unbalanced-disk-usage-with-es-2-4-x/108591):
>
> Hi, On a 7 nodes cluster, I had to replace two of the nodes. For that I used cluster routing allocation exclusion lists to make shards be moved away of those two nodes in order to shutdown ES on both. Then I added new nodes to replace them, with same settings (same name, same IP, etc), and set the cluster routing allocation exclusion parameters to empty in order to allow it to move shards to the new nodes. ES indeed moved shards to them, but one of the new nodes has too much data on it while o…

but he solved it by adding two new nodes. I may not have the luxury of being able to do this in this circumstance.

---

<div class="post-metadata">

### Author: ![loren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/loren/32/44942_2.png) [@loren](https://discuss.elastic.co/u/loren)
#### Post date: [March 22, 2018, 1:35pm UTC](https://discuss.elastic.co/t/unbalanced-disk-usage-with-es-6-1-3/125193/2 "2018-03-22T13:35:06Z")

</div>

Those 3 nodes appear to be running out of of disk, so I'd assume you've hit the default [watermarks](https://www.elastic.co/guide/en/elasticsearch/reference/master/disk-allocator.html) and Elasticsearch will no longer route shards or allocate primaries to those nodes. Check your master node logs to see if that's the case.

---

<div class="post-metadata">

### Author: ![CB193](https://avatars.discourse-cdn.com/v4/letter/c/c77e96/32.png) [@CB193](https://discuss.elastic.co/u/CB193)
#### Post date: [March 22, 2018, 2:16pm UTC](https://discuss.elastic.co/t/unbalanced-disk-usage-with-es-6-1-3/125193/3 "2018-03-22T14:16:47Z")

</div>

Yes that is correct. Those nodes are hitting the watermarks. The problem is that they are taking up 750 GB of space but the ES indexes are only 260 GB of that. So my issue is what is taking up the 500 GB of space and how to clear it up so that the nodes are no longer hitting the disk watermark.

---

<div class="post-metadata">

### Author: ![CB193](https://avatars.discourse-cdn.com/v4/letter/c/c77e96/32.png) [@CB193](https://discuss.elastic.co/u/CB193)
#### Post date: [April 3, 2018, 8:27pm UTC](https://discuss.elastic.co/t/unbalanced-disk-usage-with-es-6-1-3/125193/4 "2018-04-03T20:27:41Z")

</div>

We ended up taking the nodes out of the cluster, letting the cluster rebalance and then deleting everything in the ES data directory. When the node was added back into the cluster data was rebalanced correctly.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 1, 2018, 8:27pm UTC](https://discuss.elastic.co/t/unbalanced-disk-usage-with-es-6-1-3/125193/5 "2018-05-01T20:27:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
