# Elasticsearch complaining about space issues when there is plenty of space available **URL:** \ **Category:** Elasticsearch\ **Created:** [July 18, 2022, 5:41am UTC](https://discuss.elastic.co/t/elasticsearch-complaining-about-space-issues-when-there-is-plenty-of-space-available/309844 "2022-07-18T05:41:44Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![pushshift](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pushshift/32/50408_2.png) [@pushshift](https://discuss.elastic.co/u/pushshift)\ **Post date:** [July 18, 2022, 5:41am UTC](https://discuss.elastic.co/t/elasticsearch-complaining-about-space-issues-when-there-is-plenty-of-space-available/309844/1 "2022-07-18T05:41:44Z")
I received the following error this evening while indexing a large dataset: ```auto WARNING:root:{'create': {'_index': 'fh_hot_en_2021-05-05', '_id': '1390039787566821378', 'status': 429, 'error': {'type': 'cluster_block_exception', 'reason': 'index [fh_hot_en_2021-05-05] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];'}}} ``` However, looking at disk allocation, there is plenty of space available on all nodes: ```auto shards disk.indices disk.used disk.avail disk.total disk.percent host ip node 33 736.3gb 751gb 2.9tb 3.6tb 20 192.168.1.230 192.168.1.230 dreamcast 33 766.7gb 964.7gb 6.3tb 7.2tb 13 192.168.1.177 192.168.1.177 nintendo 32 760.3gb 773.9gb 2.9tb 3.6tb 20 192.168.1.248 192.168.1.248 hx90-1 32 752.8gb 857.9gb 4.6tb 5.4tb 15 192.168.1.202 192.168.1.202 sega 32 696.5gb 1.7tb 5.4tb 7.1tb 23 192.168.1.191 192.168.1.191 hx90-3 32 692.8gb 4.2tb 4.6tb 8.9tb 47 192.168.1.172 192.168.1.172 hx90-2 32 720.4gb 736.8gb 6.5tb 7.2tb 9 192.168.1.244 192.168.1.244 hx90-4 ``` I don't want to disable disk threshold alerts / warnings but I'm curious why Elasticsearch would think one of the disks is running low on space when the node with largest utilization is still under 50% disk usage? Can something else trigger this? Thanks! ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 18, 2022, 5:42am UTC](https://discuss.elastic.co/t/elasticsearch-complaining-about-space-issues-when-there-is-plenty-of-space-available/309844/2 "2022-07-18T05:42:23Z")
Do the logs on the master node show anything related to this? ---
**Author:** ![pushshift](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pushshift/32/50408_2.png) [@pushshift](https://discuss.elastic.co/u/pushshift)\ **Post date:** [July 18, 2022, 5:50am UTC](https://discuss.elastic.co/t/elasticsearch-complaining-about-space-issues-when-there-is-plenty-of-space-available/309844/3 "2022-07-18T05:50:51Z")
Yes -- indeed there are. There are several lines: `[2022-07-18T01:46:46,076][WARN][o.e.c.r.a.DiskThresholdMonitor] [hx90-3] flood stage disk watermark [95%] exceeded on [9dIOER48QDOB1NBks8CNeA][hx90-2][/var/lib/elasticsearch] free: 56.3gb[1.5%], all indices on this node will be marked read-only` `[2022-07-18T01:47:46,085][WARN][o.e.c.r.a.DiskThresholdMonitor] [hx90-3] flood stage disk watermark [95%] exceeded on [9dIOER48QDOB1NBks8CNeA][hx90-2][/var/lib/elasticsearch] free: 53.2gb[1.4%], all indices on this node will be marked read-only` What's really weird is that it appears to be complaining that two different nodes, each with plenty of space, are down to the same amount of low disk space. I'm going to check those two nodes to see if I can find anything additional. I've never encountered this issue before (Well I have, when disk space was actually getting low. 😊 ---
**Author:** ![pushshift](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pushshift/32/50408_2.png) [@pushshift](https://discuss.elastic.co/u/pushshift)\ **Post date:** [July 18, 2022, 5:55am UTC](https://discuss.elastic.co/t/elasticsearch-complaining-about-space-issues-when-there-is-plenty-of-space-available/309844/4 "2022-07-18T05:55:45Z")
I have more info. For the node hx90-2, it is set up to use multiple data paths. One of the data paths is indeed out of space. However, I thought with multiple data paths, it would combine all of them into one large pool? Please see below: `path.data: ['/var/lib/elasticsearch', '/elasticsearch/data', '/elasticsearch/data2']` ```auto root@HX90-2:/home/jbaumgartner# df Filesystem 1K-blocks Used Available Use% Mounted on tmpfs 6526072 1984 6524088 1% /run /dev/nvme0n1p2 3844033552 3791090700 52145172 99% / tmpfs 32630340 0 32630340 0% /dev/shm tmpfs 5120 4 5116 1% /run/lock /dev/sdb 1921725720 27689140 1796344468 2% /elasticsearch/data2 /dev/sda 3844551496 701724952 3142419460 19% /elasticsearch/data /dev/nvme0n1p1 523248 5364 517884 2% /boot/efi tmpfs 6526068 76 6525992 1% /run/user/127 tmpfs 6526068 64 6526004 1% /run/user/1000 s3fs 18014398509465600 0 18014398509465600 0% /mnt/wasabi ``` So Elasticsearch is right in seeing that one of the data paths is almost full, but then my assumptions about how multiple data paths is used must be incorrect because I thought they would be used as one large pool. However internally, I guess it makes sense if shard data is already on one path and that path runs out of space (but shouldn't it just move the shard over to a path with more space?) ---
**Author:** ![pushshift](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pushshift/32/50408_2.png) [@pushshift](https://discuss.elastic.co/u/pushshift)\ **Post date:** [July 18, 2022, 5:59am UTC](https://discuss.elastic.co/t/elasticsearch-complaining-about-space-issues-when-there-is-plenty-of-space-available/309844/5 "2022-07-18T05:59:44Z")
One more addition: If multiple data paths does not work like this, then the information from /\_cat/allocation might need to include more information for nodes with multiple data paths, like showing a breakdown of each data path and what the allocation is per data path -- because as of right now, it is showing the aggregate of the sum of all data paths for space left across all of them. That means someone could look at allocation and think the cluster is fine but there could be a data path dangerously low on space. Side note: Is using multiple data paths frowned upon or more of a sysadmin PITA? Should I be using LVM or some other RAID type structure for a node with multiple disks instead of using multiple data paths? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 18, 2022, 7:34am UTC](https://discuss.elastic.co/t/elasticsearch-complaining-about-space-issues-when-there-is-plenty-of-space-available/309844/6 "2022-07-18T07:34:35Z")
> [@pushshift](#): > > I thought with multiple data paths, it would combine all of them into one large pool It does not, no. Please see [Important Elasticsearch configuration | Elasticsearch Guide [8.3] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/8.3/important-settings.html#_multiple_data_paths). > [@pushshift](#): > > If multiple data paths does not work like this, then the information from /\_cat/allocation might need to include more information for nodes with multiple data paths The `_cat` APIs are intended to be high level, summaries of what is happening. You can raise a feature request to see if it can be altered, but please keep that in mind. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 18, 2022, 8:02am UTC](https://discuss.elastic.co/t/elasticsearch-complaining-about-space-issues-when-there-is-plenty-of-space-available/309844/7 "2022-07-18T08:02:20Z")
> [@pushshift](#): > > Is using multiple data paths frowned upon or more of a sysadmin PITA? Should I be using LVM or some other RAID type structure for a node with multiple disks instead of using multiple data paths? Yes. Multiple data paths doesn't work as you expect, and it's deprecated. The recommended setup is to use a single data path per node, possibly running multiple nodes per host and/or combining volumes together using something like RAID. ---
**Author:** ![pushshift](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pushshift/32/50408_2.png) [@pushshift](https://discuss.elastic.co/u/pushshift)\ **Post date:** [July 19, 2022, 5:11am UTC](https://discuss.elastic.co/t/elasticsearch-complaining-about-space-issues-when-there-is-plenty-of-space-available/309844/8 "2022-07-19T05:11:09Z")
Thank you David! This is great advice. We'll make the necessary changes. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 16, 2022, 5:11am UTC](https://discuss.elastic.co/t/elasticsearch-complaining-about-space-issues-when-there-is-plenty-of-space-available/309844/9 "2022-08-16T05:11:24Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.