# Disk space issues on 7.10

**URL:** <https://discuss.elastic.co/t/disk-space-issues-on-7-10/279584>\
**Category:** Elasticsearch\
**Created:** [July 26, 2021, 3:34am UTC](https://discuss.elastic.co/t/disk-space-issues-on-7-10/279584 "2021-07-26T03:34:56Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Russell\_Fulton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russell_fulton/32/62888_2.png) [@Russell\_Fulton](https://discuss.elastic.co/u/Russell_Fulton)\
**Post date:** [July 26, 2021, 3:34am UTC](https://discuss.elastic.co/t/disk-space-issues-on-7-10/279584/1 "2021-07-26T03:34:56Z")

</div>

I have two more or less identical servers in my cluster that each have 3 disks in the data\_path each with 1.5TB.  
Recently one of the server has started running out of disk on one of the three disks (the first in the list if it matters).

Over all there is plenty of disk space so the high water mark is never reached ES just suddenly runs out of space on one disk then and shuts down.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 26, 2021, 5:30am UTC](https://discuss.elastic.co/t/disk-space-issues-on-7-10/279584/2 "2021-07-26T05:30:03Z")

</div>

At a guess, that might come down to the differing shard sizes on the paths.

---

<div class="post-metadata">

**Author:** ![Russell\_Fulton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russell_fulton/32/62888_2.png) [@Russell\_Fulton](https://discuss.elastic.co/u/Russell_Fulton)\
**Post date:** [July 26, 2021, 7:03am UTC](https://discuss.elastic.co/t/disk-space-issues-on-7-10/279584/3 "2021-07-26T07:03:49Z")

</div>

Thanks Mark, how would I check?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 26, 2021, 7:24am UTC](https://discuss.elastic.co/t/disk-space-issues-on-7-10/279584/4 "2021-07-26T07:24:30Z")

</div>

Is shard allocation equal on all nodes?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [July 26, 2021, 8:54am UTC](https://discuss.elastic.co/t/disk-space-issues-on-7-10/279584/5 "2021-07-26T08:54:51Z")

</div>

> [@Russell\_Fulton](#):
>
> Over all there is plenty of disk space so the high water mark is never reached ES just suddenly runs out of space on one disk then and shuts down.

This seems weird for a couple of reasons:

1. the disk watermarks are per-node things: if one node starts to run out of space then it will hit its watermark, even if there's plenty of space overall.

2. when a node runs out of space it doesn't shut down.

Can you clarify what you mean exactly, with logs and so on? We only check disk space every 30s or so, so if you write a huge amount of data in 30s and/or the watermarks are set very close then it is possible to go from below the high watermark to a full disk before any protection mechanisms kick in.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 23, 2021, 8:55am UTC](https://discuss.elastic.co/t/disk-space-issues-on-7-10/279584/6 "2021-08-23T08:55:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.