# ECK Elasticsearch at 100% disk usage
**URL:** https://discuss.elastic.co/t/eck-elasticsearch-at-100-disk-usage/372737
**Category:** Elastic Cloud on Kubernetes (ECK)
**Created:** [January 3, 2025, 9:47am UTC](https://discuss.elastic.co/t/eck-elasticsearch-at-100-disk-usage/372737 "2025-01-03T09:47:12Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@lduvnjak](https://discuss.elastic.co/u/lduvnjak)
#### Post date: [January 3, 2025, 9:47am UTC](https://discuss.elastic.co/t/eck-elasticsearch-at-100-disk-usage/372737/1 "2025-01-03T09:47:12Z")
Hi Everyone,
We've been having some issues with our Elasticsearch cluster hitting 100% disk usage on one or more nodes.
It is a three node cluster with two master/data nodes and one voting-only node.
The nodes have a PV created by the Local Volume Provisioner storage class.
```auto
[root@k8s02 storage]# df -h /var/elasticsearch/storage/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/almalinux-elastic 1019G 677G 342G 67% /var/elasticsearch/storage
[root@k8s02 storage]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
local-pv-7db3cb5 969Gi RWO Delete Bound default/elasticsearch-data-elastic-es-master-0 local-storage 170d
local-pv-9a836200 969Gi RWO Delete Bound default/elasticsearch-data-elastic-es-master-1 local-storage 170d
```
We have about 10+ clusters in production. In every other instance, when hitting the flood stage all the indices get the `read_only_allow_delete` option set accordingly, except for this one.
There are no notable logs on any of the ELK stack components, nor on the OS of the nodes running Kubernetes. This has happened to this cluster a few times already, forcing us to increase the disk space each time.
Does anyone have any clue on what the reason for that could be?
Here are all the cluster settings the node has configured:
```auto
[2025-01-03T09:36:57,957][INFO][o.e.c.s.ClusterSettings] [elastic-es-master-1] updating [cluster.routing.allocation.node_concurrent_incoming_recoveries] from [2] to [4]
[2025-01-03T09:36:57,957][INFO][o.e.c.s.ClusterSettings] [elastic-es-master-1] updating [cluster.routing.allocation.disk.watermark.high] from [90%] to [94%]
[2025-01-03T09:36:57,958][INFO][o.e.c.s.ClusterSettings] [elastic-es-master-1] updating [cluster.routing.allocation.disk.watermark.flood_stage] from [95%] to [97%]
[2025-01-03T09:36:57,958][INFO][o.e.c.s.ClusterSettings] [elastic-es-master-1] updating [cluster.routing.allocation.disk.watermark.high.max_headroom] from [150GB] to [-1]
[2025-01-03T09:36:57,958][INFO][o.e.c.s.ClusterSettings] [elastic-es-master-1] updating [cluster.routing.allocation.disk.watermark.flood_stage.max_headroom] from [100GB] to [-1]
[2025-01-03T09:36:57,958][INFO][o.e.c.s.ClusterSettings] [elastic-es-master-1] updating [cluster.max_shards_per_node] from [1000] to [3000]
[2025-01-03T09:36:57,959][INFO][o.e.c.s.ClusterSettings] [elastic-es-master-1] updating [cluster.routing.allocation.disk.watermark.low] from [85%] to [92%]
[2025-01-03T09:36:57,959][INFO][o.e.c.s.ClusterSettings] [elastic-es-master-1] updating [cluster.routing.allocation.disk.watermark.low.max_headroom] from [200GB] to [-1]
[2025-01-03T09:36:57,959][INFO][o.e.c.s.ClusterSettings] [elastic-es-master-1] updating [cluster.routing.allocation.disk.watermark.high] from [90%] to [94%]
[2025-01-03T09:36:57,959][INFO][o.e.c.s.ClusterSettings] [elastic-es-master-1] updating [cluster.routing.allocation.disk.watermark.high.max_headroom] from [150GB] to [-1]
[2025-01-03T09:36:57,959][INFO][o.e.c.s.ClusterSettings] [elastic-es-master-1] updating [cluster.routing.allocation.disk.watermark.flood_stage] from [95%] to [97%]
[2025-01-03T09:36:57,959][INFO][o.e.c.s.ClusterSettings] [elastic-es-master-1] updating [cluster.routing.allocation.disk.watermark.flood_stage.max_headroom] from [100GB] to [-1]
[2025-01-03T09:36:57,960][INFO][o.e.c.s.ClusterSettings] [elastic-es-master-1] updating [cluster.routing.allocation.disk.watermark.low] from [85%] to [92%]
[2025-01-03T09:36:57,960][INFO][o.e.c.s.ClusterSettings] [elastic-es-master-1] updating [cluster.routing.allocation.disk.watermark.low.max_headroom] from [200GB] to [-1]
[2025-01-03T09:36:57,960][INFO][o.e.c.s.ClusterSettings] [elastic-es-master-1] updating [cluster.routing.allocation.disk.watermark.high] from [90%] to [94%]
[2025-01-03T09:36:57,960][INFO][o.e.c.s.ClusterSettings] [elastic-es-master-1] updating [cluster.routing.allocation.disk.watermark.high.max_headroom] from [150GB] to [-1]
[2025-01-03T09:36:57,960][INFO][o.e.c.s.ClusterSettings] [elastic-es-master-1] updating [cluster.routing.allocation.disk.watermark.flood_stage] from [95%] to [97%]
[2025-01-03T09:36:57,960][INFO][o.e.c.s.ClusterSettings] [elastic-es-master-1] updating [cluster.routing.allocation.disk.watermark.flood_stage.max_headroom] from [100GB] to [-1]
```
Any help is greatly appreciated!
Cheers,
Luka
---
### Author:  [@Musab\_Dogan](https://discuss.elastic.co/u/Musab_Dogan)
#### Post date: [January 5, 2025, 3:30pm UTC](https://discuss.elastic.co/t/eck-elasticsearch-at-100-disk-usage/372737/2 "2025-01-05T15:30:55Z")
Can you share the output of the following API call result?
```auto
GET _cat/allocation?v
```
---
### Author:  [@RainTown](https://discuss.elastic.co/u/RainTown)
#### Post date: [January 11, 2025, 12:25am UTC](https://discuss.elastic.co/t/eck-elasticsearch-at-100-disk-usage/372737/3 "2025-01-11T00:25:04Z")
@lduvnjak any update here?
If you resolved your issue, and I hope you did, always helps the community to share how you did it!
---
### Author:  [@lduvnjak](https://discuss.elastic.co/u/lduvnjak)
#### Post date: [January 14, 2025, 1:31pm UTC](https://discuss.elastic.co/t/eck-elasticsearch-at-100-disk-usage/372737/4 "2025-01-14T13:31:56Z")
Sorry for the late reply, for some reason I didn't get an email that someone replied in the thread.
Here's the output:
```auto
shards shards.undesired write_load.forecast disk.indices.forecast disk.indices disk.used disk.avail disk.total disk.percent host ip node node.role
281 0 0.0 794.6gb 764.6gb 772.7gb 245.2gb 1018gb 75 10.233.73.54 10.233.73.54 elastic-es-master-0 cdfhilmrstw
281 0 0.0 794.6gb 761.6gb 768.9gb 249gb 1018gb 75 10.233.112.62 10.233.112.62 elastic-es-master-1 cdfhilmrstw
```
The cluster is working fine ATM since we forcefully restarted the stuck pod.
Also, we manually filled up the disk with `fallocate` and it detected it just fine, triggering the low, high, and flood stages accordingly. So we're not really sure when or why this happens.
---
### Author:  [@Musab\_Dogan](https://discuss.elastic.co/u/Musab_Dogan)
#### Post date: [January 14, 2025, 1:44pm UTC](https://discuss.elastic.co/t/eck-elasticsearch-at-100-disk-usage/372737/5 "2025-01-14T13:44:05Z")
All looks good. Use some alerting system like AutoOps to make sure your cluster is healthy and the disks are not full.
> **[AutoOps: Simplify cluster management](https://www.elastic.co/platform/autoops)**
>
> Discover how AutoOps for Elastic simplifies cluster management with performance insights, cost optimization, and real-time issue detection. Try it free with Elastic Cloud Hosted....
---
### Author:  [@RainTown](https://discuss.elastic.co/u/RainTown)
#### Post date: [January 15, 2025, 11:27pm UTC](https://discuss.elastic.co/t/eck-elasticsearch-at-100-disk-usage/372737/6 "2025-01-15T23:27:14Z")
There’s really not enough information here to diagnose why _sometimes_ things don’t work as you expect.
If I’ve understood correctly, if you create 100% disk usage outside Elasticsearch, eg using fallioate, then the alarms are triggered both as expected and as on other clusters you manage.
But on this cluster alone, if Elasticsearch itself eats up all the available disk space, you don’t get the log entries for low/high/flood.
You could increase the specific classes logging to DEBUG or TRACE , though someone else would need to advise the precise names to use.
---
### Author:  [@lduvnjak](https://discuss.elastic.co/u/lduvnjak)
#### Post date: [January 16, 2025, 8:16am UTC](https://discuss.elastic.co/t/eck-elasticsearch-at-100-disk-usage/372737/7 "2025-01-16T08:16:53Z")
Agreed, it's not really reproducible manually so it's almost impossible to troubleshoot.
We're removing this cluster anyways so let's hope it's a niche bug on this specific installation and won't happen again.
Thanks for the help!