Elastic 5.4 - watermarks and multiple disks issues

Elastic Stack Elasticsearch
1

All: I searched this list and only found one message from 2015 related to this and it was not answered.

I have an elasticsearch 5.4 cluster. It has 20 data nodes with 5 of them also serving as master
nodes. Each node has two 512GB disks for data (/data01 and /data02).
The following is the settings of one node:

[oe_admin@oedevnode26 ~]$ curl -k --cert "/etc/pki/java/temp.pem" -s -XGET "https://oedevnode09:9201/_nodes?pretty&filter_path=nodes.*.settings.path.data"
{
"nodes" : {
"cr1kHpTGRQmZSj59gs9Bcw" : {
"settings" : {
"path" : {
"data" : [
"/data01/elasticsearch/oedl/data",
"/data02/elasticsearch/oedl/data"
]
}
}
},

The issue is that elastic is not balancing the data across both disks. So /data01 is filling up and exceeding 80% which is triggering a warning from my nagios monitoring software while /data02 has minimal data (ES shares these disks with hadoop hdfs). Here is what a typical DF looks like on a node:

/dev/sdd1 512G 74G 439G 15% /data02
/dev/sdc1 512G 408G 105G 80% /data01

My watermark settings are as follows:

[oe_admin@oedevnode26 ~]$ curl -k --cert "/etc/pki/java/temp.pem" -s -XGET "https://oedevnode09:9201/_cluster/settings?pretty=true"
{
"persistent" : { },
"transient" : {
"cluster" : {
"routing" : {
"allocation" : {
"disk" : {
"watermark" : {
"low" : "80%",
"high" : "75%"
}
}
}
}
}
}
}
I have tried the default settings and the goal of these was to get ES to move data to the second disk.

It appears that the watermark settings are being applied to the combined disk space versus each individual disk.

Is there are way to force ES to balance the space used across both disks?
Any easy ways to even balance things manually?
If I switch the order of the disks in the path will that change the order that ES writes to the disks?

Sorry if a missed any post already here that answers this problem.

2

Elasticsearch doesn't balance between disks. If you need this functionality, you can either use a filesystem that spans both disks (e.g. RAID0, LVM) or else you can run one node per disk.

3

David, thanks. That is what I was worried about.
On some of my nodes it is the first disk in the
list that is filling up, on others it is the second?
How does elastic decide which disk in the data path to use?
Is it some sort of round robin per shard?

This is all in the cloud so having a single large disk is not an issue.
But I would have to find some way to relocate all the data off the least used disk to
a new bigger one. Of course I will also need to do the same with my Hadoop data.

4

It's probably safest to consider it to be undefined which path is used.

Can you attach another single volume large enough for all your data? If so I would do that, run a new node on that single volume, then use shard allocation filtering to vacate the old node.

5

It'd probably be easier to just add an entirely new node, then move the data over.

You really only need 3 master nodes.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.