# Understanding Disk-based Shard Allocation better **URL:** \ **Category:** Elasticsearch\ **Created:** [February 10, 2019, 10:00am UTC](https://discuss.elastic.co/t/understanding-disk-based-shard-allocation-better/167746 "2019-02-10T10:00:52Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![kghbln](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kghbln/32/40763_2.png) [@kghbln](https://discuss.elastic.co/u/kghbln)\ **Post date:** [February 10, 2019, 10:00am UTC](https://discuss.elastic.co/t/understanding-disk-based-shard-allocation-better/167746/1 "2019-02-10T10:00:52Z")
Even after reading the docu I have no idea what to set to my "elasticsearch.yml" file to prevent the following pain which I am getting all at the same time: low disk watermark [20gb] exceeded on ... high disk watermark [10gb] exceeded on ... flood stage disk watermark [5gb] exceeded on ... My hard disk has 25 GB of free space available and is at 93%. This setting did not work: ``` cluster.routing.allocation.disk.watermark.flood_stage: 99% cluster.routing.allocation.disk.watermark.low: 95% cluster.routing.allocation.disk.watermark.high: 97% ``` This setting did not work either (see above): ``` cluster.routing.allocation.disk.watermark.flood_stage: 5gb cluster.routing.allocation.disk.watermark.low: 20gb cluster.routing.allocation.disk.watermark.high: 10gb ``` I consider the amount of data I would like to move to Elasticsearch as marginal. Any insights will be great. I guess I just completely misunderstood that the docu is trying to tell me. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [February 10, 2019, 11:30am UTC](https://discuss.elastic.co/t/understanding-disk-based-shard-allocation-better/167746/2 "2019-02-10T11:30:33Z")
> [@kghbln](#): > > My hard disk has 25 GB of free space available and is at 93%. If so, the settings you quote look correct. Can you share the log messages you're seeing indicating that the disk-based shard allocator is causing a problem? ---
**Author:** ![kghbln](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kghbln/32/40763_2.png) [@kghbln](https://discuss.elastic.co/u/kghbln)\ **Post date:** [February 10, 2019, 11:55am UTC](https://discuss.elastic.co/t/understanding-disk-based-shard-allocation-better/167746/3 "2019-02-10T11:55:53Z")
> [@DavidTurner](#): > > Can you share the log messages you're seeing indicating that the disk-based shard allocator is causing a problem? Sure, see the [error log](https://gist.githubusercontent.com/kghbln/7759cd3060b06a2d7b637589faf8b61e/raw/b0b08648c060e959732cbf710652c644db79cc50/discuss.elastic.co:167746) ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [February 10, 2019, 12:18pm UTC](https://discuss.elastic.co/t/understanding-disk-based-shard-allocation-better/167746/4 "2019-02-10T12:18:34Z")
```auto [2019-02-10T05:32:43,078][WARN][o.e.c.r.a.DiskThresholdMonitor] [Ql8wRCh] flood stage disk watermark [5gb] exceeded on [Ql8wRChKRb-YoZ0gz0nBDA][Ql8wRCh][/var/lib/elasticsearch/nodes/0] free: 2.4gb[0.8%], all indices on this node will be marked read-only ``` This indicates that you have substantially less free space than you think. Can you run `df` and see if it agrees with these numbers? ---
**Author:** ![kghbln](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kghbln/32/40763_2.png) [@kghbln](https://discuss.elastic.co/u/kghbln)\ **Post date:** [February 10, 2019, 12:21pm UTC](https://discuss.elastic.co/t/understanding-disk-based-shard-allocation-better/167746/5 "2019-02-10T12:21:12Z")
> [@DavidTurner](#): > > This indicates that you have substantially less free space than you think. Can you run `df` and see if it agrees with these numbers? This is exactly what is puzzling me: Just did a `df -h` and there appears to be enough space to cater for the setting. ``` Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf udev 992M 0 992M 0% /dev tmpfs 201M 22M 179M 11% /run /dev/vda1 292G 253G 25G 92% / tmpfs 1003M 0 1003M 0% /dev/shm tmpfs 5,0M 0 5,0M 0% /run/lock tmpfs 1003M 0 1003M 0% /sys/fs/cgroup tmpfs 201M 0 201M 0% /run/user/0 ``` Note: I am aware that I do need more disk space. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [February 10, 2019, 12:26pm UTC](https://discuss.elastic.co/t/understanding-disk-based-shard-allocation-better/167746/6 "2019-02-10T12:26:27Z")
Is Elasticsearch still running and reporting a lack of space every 30 seconds? The timestamp on the last log message you shared looks several hours old (or maybe is in a different timezone). ---
**Author:** ![kghbln](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kghbln/32/40763_2.png) [@kghbln](https://discuss.elastic.co/u/kghbln)\ **Post date:** [February 10, 2019, 12:41pm UTC](https://discuss.elastic.co/t/understanding-disk-based-shard-allocation-better/167746/7 "2019-02-10T12:41:45Z")
Actually no. I am getting this every morning for a couple of days now. When I mitigate by unblocking with curl: ``` curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}' ``` This however makes me think that it is a temporary space shortage at night, when I do backups. Hmm ... let me change something and get back here tomorrow. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [February 10, 2019, 12:54pm UTC](https://discuss.elastic.co/t/understanding-disk-based-shard-allocation-better/167746/8 "2019-02-10T12:54:25Z")
> [@kghbln](#): > > This however makes me think that it is a temporary space shortage at night, when I do backups That sounds about right. _Something_ (possibly Elasticsearch, possibly another process) is consuming nearly all your disk space and then releasing it again. Elasticsearch is rather averse to running out of disk space, so when you exceed the `flood_stage` watermark it takes the fairly drastic step of marking everything as read-only to protect itself. In the long run you will need more disk space. You may be able to work around this in the short term by isolating Elasticsearch on its own filesystem (assuming it's not Elasticsearch that's consuming all the disk). If desperate you can set the `flood_stage` watermark to something tiny (e.g. `100mb`). ---
**Author:** ![kghbln](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kghbln/32/40763_2.png) [@kghbln](https://discuss.elastic.co/u/kghbln)\ **Post date:** [February 10, 2019, 1:05pm UTC](https://discuss.elastic.co/t/understanding-disk-based-shard-allocation-better/167746/9 "2019-02-10T13:05:39Z")
Thanks a lot for your insight. > [@DavidTurner](#): > > In the long run you will need more disk space. Indeed and I am already planning for a new server. > [@DavidTurner](#): > > You may be able to work around this in the short term by isolating Elasticsearch on its own filesystem (assuming it's not Elasticsearch that's consuming all the disk). If desperate you can set the `flood_stage` watermark to something tiny (e.g. `100mb` ). I will try this after I got the results from what I just changed concerning the backups. Thanks for the tip! ---
**Author:** ![kghbln](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kghbln/32/40763_2.png) [@kghbln](https://discuss.elastic.co/u/kghbln)\ **Post date:** [February 11, 2019, 9:58am UTC](https://discuss.elastic.co/t/understanding-disk-based-shard-allocation-better/167746/10 "2019-02-11T09:58:29Z")
> [@kghbln](#): > > I will try this after I got the results from what I just changed concerning the backups. Thanks for the tip! Things cannot be solved on my server since even after tweaking there is not enough disk space left to avoid high and low disk watermark warnings/issues. I need to get a bigger volume with sufficient space. However setting `flood_stage` to a much lower value for the time being helps preventing the rerouting of shards from being stopped entirely due to locking down the disk for Elasticsearch: ``` cluster.routing.allocation.disk.watermark.flood_stage: 1gb ``` Other lessons learned: Elasticsearch actually needs quite a lot for disk space to do replication so the requirements in GB suggested for high and low disk watermarks do need to be met. If you have disk space galore one can use settings with percentages and if you are tight in disk space you need to set hard values in gb. @DavidTurner Thanks a lot for guiding me through my troubled waters. This is very much appreciated! ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [February 11, 2019, 2:45pm UTC](https://discuss.elastic.co/t/understanding-disk-based-shard-allocation-better/167746/11 "2019-02-11T14:45:43Z")
You're welcome @kghbln, glad we could help. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 11, 2019, 2:49pm UTC](https://discuss.elastic.co/t/understanding-disk-based-shard-allocation-better/167746/12 "2019-03-11T14:49:37Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.