Elasticsearch complaining about space issues when there is plenty of space available

Elastic Stack Elasticsearch
1

I received the following error this evening while indexing a large dataset:

WARNING:root:{'create': {'_index': 'fh_hot_en_2021-05-05', '_id': '1390039787566821378', 'status': 429, 'error': {'type': 'cluster_block_exception', 'reason': 'index 
[fh_hot_en_2021-05-05] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];'}}}

However, looking at disk allocation, there is plenty of space available on all nodes:

shards disk.indices disk.used disk.avail disk.total disk.percent host          ip            node
    33      736.3gb     751gb      2.9tb      3.6tb           20 192.168.1.230 192.168.1.230 dreamcast
    33      766.7gb   964.7gb      6.3tb      7.2tb           13 192.168.1.177 192.168.1.177 nintendo
    32      760.3gb   773.9gb      2.9tb      3.6tb           20 192.168.1.248 192.168.1.248 hx90-1
    32      752.8gb   857.9gb      4.6tb      5.4tb           15 192.168.1.202 192.168.1.202 sega
    32      696.5gb     1.7tb      5.4tb      7.1tb           23 192.168.1.191 192.168.1.191 hx90-3
    32      692.8gb     4.2tb      4.6tb      8.9tb           47 192.168.1.172 192.168.1.172 hx90-2
    32      720.4gb   736.8gb      6.5tb      7.2tb            9 192.168.1.244 192.168.1.244 hx90-4

I don't want to disable disk threshold alerts / warnings but I'm curious why Elasticsearch would think one of the disks is running low on space when the node with largest utilization is still under 50% disk usage? Can something else trigger this?

Thanks!

2

Do the logs on the master node show anything related to this?

3

Yes -- indeed there are. There are several lines:

[2022-07-18T01:46:46,076][WARN ][o.e.c.r.a.DiskThresholdMonitor] [hx90-3] flood stage disk watermark [95%] exceeded on [9dIOER48QDOB1NBks8CNeA][hx90-2][/var/lib/elasticsearch] free: 56.3gb[1.5%], all indices on this node will be marked read-only

[2022-07-18T01:47:46,085][WARN ][o.e.c.r.a.DiskThresholdMonitor] [hx90-3] flood stage disk watermark [95%] exceeded on [9dIOER48QDOB1NBks8CNeA][hx90-2][/var/lib/elasticsearch] free: 53.2gb[1.4%], all indices on this node will be marked read-only

What's really weird is that it appears to be complaining that two different nodes, each with plenty of space, are down to the same amount of low disk space. I'm going to check those two nodes to see if I can find anything additional. I've never encountered this issue before (Well I have, when disk space was actually getting low. :blush:

4

I have more info. For the node hx90-2, it is set up to use multiple data paths. One of the data paths is indeed out of space. However, I thought with multiple data paths, it would combine all of them into one large pool? Please see below:

path.data: ['/var/lib/elasticsearch', '/elasticsearch/data', '/elasticsearch/data2']

root@HX90-2:/home/jbaumgartner# df
Filesystem             1K-blocks       Used         Available Use% Mounted on
tmpfs                    6526072       1984           6524088   1% /run
/dev/nvme0n1p2        3844033552 3791090700          52145172  99% /
tmpfs                   32630340          0          32630340   0% /dev/shm
tmpfs                       5120          4              5116   1% /run/lock
/dev/sdb              1921725720   27689140        1796344468   2% /elasticsearch/data2
/dev/sda              3844551496  701724952        3142419460  19% /elasticsearch/data
/dev/nvme0n1p1            523248       5364            517884   2% /boot/efi
tmpfs                    6526068         76           6525992   1% /run/user/127
tmpfs                    6526068         64           6526004   1% /run/user/1000
s3fs           18014398509465600          0 18014398509465600   0% /mnt/wasabi

So Elasticsearch is right in seeing that one of the data paths is almost full, but then my assumptions about how multiple data paths is used must be incorrect because I thought they would be used as one large pool. However internally, I guess it makes sense if shard data is already on one path and that path runs out of space (but shouldn't it just move the shard over to a path with more space?)

5

One more addition:

If multiple data paths does not work like this, then the information from /_cat/allocation might need to include more information for nodes with multiple data paths, like showing a breakdown of each data path and what the allocation is per data path -- because as of right now, it is showing the aggregate of the sum of all data paths for space left across all of them. That means someone could look at allocation and think the cluster is fine but there could be a data path dangerously low on space.

Side note: Is using multiple data paths frowned upon or more of a sysadmin PITA? Should I be using LVM or some other RAID type structure for a node with multiple disks instead of using multiple data paths?

6

It does not, no. Please see Important Elasticsearch configuration | Elasticsearch Guide [8.3] | Elastic.

The _cat APIs are intended to be high level, summaries of what is happening. You can raise a feature request to see if it can be altered, but please keep that in mind.

7

Yes. Multiple data paths doesn't work as you expect, and it's deprecated. The recommended setup is to use a single data path per node, possibly running multiple nodes per host and/or combining volumes together using something like RAID.

2 Likes
8

Thank you David! This is great advice. We'll make the necessary changes.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.