# Elastic 5.4 - watermarks and multiple disks issues

**URL:** <https://discuss.elastic.co/t/elastic-5-4-watermarks-and-multiple-disks-issues/166826>\
**Category:** Elasticsearch\
**Created:** [February 2, 2019, 4:16pm UTC](https://discuss.elastic.co/t/elastic-5-4-watermarks-and-multiple-disks-issues/166826 "2019-02-02T16:16:37Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![bytemedwb](https://avatars.discourse-cdn.com/v4/letter/b/7bcc69/32.png) [@bytemedwb](https://discuss.elastic.co/u/bytemedwb)\
**Post date:** [February 2, 2019, 4:16pm UTC](https://discuss.elastic.co/t/elastic-5-4-watermarks-and-multiple-disks-issues/166826/1 "2019-02-02T16:16:37Z")

</div>

All: I searched this list and only found one message from 2015 related to this and it was not answered.

I have an elasticsearch 5.4 cluster. It has 20 data nodes with 5 of them also serving as master  
nodes. Each node has two 512GB disks for data (/data01 and /data02).  
The following is the settings of one node:

> [oe\_admin@oedevnode26 ~]$ curl -k --cert "/etc/pki/java/temp.pem" -s -XGET "[https://oedevnode09:9201/\_nodes?pretty&filter\_path=nodes.\*.settings.path.data](https://oedevnode09:9201/_nodes?pretty&filter_path=nodes.*.settings.path.data)"  
> {  
> "nodes" : {  
> "cr1kHpTGRQmZSj59gs9Bcw" : {  
> "settings" : {  
> "path" : {  
> "data" : [  
> "/data01/elasticsearch/oedl/data",  
> "/data02/elasticsearch/oedl/data"  
> ]  
> }  
> }  
> },

The issue is that elastic is not balancing the data across both disks. So /data01 is filling up and exceeding 80% which is triggering a warning from my nagios monitoring software while /data02 has minimal data (ES shares these disks with hadoop hdfs). Here is what a typical DF looks like on a node:

> /dev/sdd1 512G 74G 439G 15% /data02  
> /dev/sdc1 512G 408G 105G 80% /data01

My watermark settings are as follows:

> [oe\_admin@oedevnode26 ~]$ curl -k --cert "/etc/pki/java/temp.pem" -s -XGET "[https://oedevnode09:9201/\_cluster/settings?pretty=true](https://oedevnode09:9201/_cluster/settings?pretty=true)"  
> {  
> "persistent" : { },  
> "transient" : {  
> "cluster" : {  
> "routing" : {  
> "allocation" : {  
> "disk" : {  
> "watermark" : {  
> "low" : "80%",  
> "high" : "75%"  
> }  
> }  
> }  
> }  
> }  
> }  
> }  
> I have tried the default settings and the goal of these was to get ES to move data to the second disk.

It appears that the watermark settings are being applied to the combined disk space versus each individual disk.

Is there are way to force ES to balance the space used across both disks?  
Any easy ways to even balance things manually?  
If I switch the order of the disks in the path will that change the order that ES writes to the disks?

Sorry if a missed any post already here that answers this problem.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 3, 2019, 6:34pm UTC](https://discuss.elastic.co/t/elastic-5-4-watermarks-and-multiple-disks-issues/166826/2 "2019-02-03T18:34:08Z")

</div>

> [@bytemedwb](#):
>
> Is there are way to force ES to balance the space used across both disks?

Elasticsearch doesn't balance between disks. If you need this functionality, you can either use a filesystem that spans both disks (e.g. RAID0, LVM) or else you can run one node per disk.

---

<div class="post-metadata">

**Author:** ![bytemedwb](https://avatars.discourse-cdn.com/v4/letter/b/7bcc69/32.png) [@bytemedwb](https://discuss.elastic.co/u/bytemedwb)\
**Post date:** [February 3, 2019, 6:59pm UTC](https://discuss.elastic.co/t/elastic-5-4-watermarks-and-multiple-disks-issues/166826/3 "2019-02-03T18:59:22Z")

</div>

David, thanks. That is what I was worried about.  
On some of my nodes it is the first disk in the  
list that is filling up, on others it is the second?  
How does elastic decide which disk in the data path to use?  
Is it some sort of round robin per shard?

This is all in the cloud so having a single large disk is not an issue.  
But I would have to find some way to relocate all the data off the least used disk to  
a new bigger one. Of course I will also need to do the same with my Hadoop data.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 3, 2019, 7:33pm UTC](https://discuss.elastic.co/t/elastic-5-4-watermarks-and-multiple-disks-issues/166826/4 "2019-02-03T19:33:36Z")

</div>

> [@bytemedwb](#):
>
> How does elastic decide which disk in the data path to use?  
> Is it some sort of round robin per shard?

It's probably safest to consider it to be undefined which path is used.

> [@bytemedwb](#):
>
> This is all in the cloud so having a single large disk is not an issue.  
> But I would have to find some way to relocate all the data off the least used disk to  
> a new bigger one.

Can you attach another single volume large enough for all your data? If so I would do that, run a new node on that single volume, then use [shard allocation filtering](https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-allocation-filtering.html) to vacate the old node.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 3, 2019, 9:45pm UTC](https://discuss.elastic.co/t/elastic-5-4-watermarks-and-multiple-disks-issues/166826/5 "2019-02-03T21:45:09Z")

</div>

> [@DavidTurner](#):
>
> Can you attach another single volume large enough for all your data? If so I would do that, run a new node on that single volume, then use [shard allocation filtering](https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-allocation-filtering.html) to vacate the old node.

It'd probably be easier to just add an entirely new node, then move the data over.

> [@bytemedwb](#):
>
> I have an elasticsearch 5.4 cluster. It has 20 data nodes with 5 of them also serving as master  
> nodes

You really only need 3 master nodes.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 3, 2019, 9:45pm UTC](https://discuss.elastic.co/t/elastic-5-4-watermarks-and-multiple-disks-issues/166826/6 "2019-03-03T21:45:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.