# DataNode disk full, despite on flood\_stage configuration **URL:** \ **Category:** Elasticsearch\ **Created:** [December 5, 2022, 12:37pm UTC](https://discuss.elastic.co/t/datanode-disk-full-despite-on-flood-stage-configuration/320468 "2022-12-05T12:37:27Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![gidonshn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gidonshn/32/114231_2.png) [@gidonshn](https://discuss.elastic.co/u/gidonshn)\ **Post date:** [December 5, 2022, 12:37pm UTC](https://discuss.elastic.co/t/datanode-disk-full-despite-on-flood-stage-configuration/320468/1 "2022-12-05T12:37:27Z")
Hi All. I have a cluster (7.16.2) running on k8s, with multiple DataNodes, dedicated ClientNodes, and Dedicated MasterNodes. Some of the DataNodes are "hot" data nodes, and some of them are "warm". I have daily indices created and all of them are managed using index templates and index lifecycles (hot-\>warm-\>delete). My watermarks are configured as default: cluster.routing.allocation.disk.watermark.low=85% cluster.routing.allocation.disk.watermark.high=90% cluster.routing.allocation.disk.watermark.flood\_stage=95% Some of my indices have replica shards and some do not. One of the "hot" DataNodes (which is a pod) got to 100% disk usage and couldn't rejoin the cluster. How can that happen? Shouldn't the flood\_stage parameter mark the indices with shards on the node as "ReadOnly" and stop all writes to the disk? Is there a configuration I somehow could have missed or misconfigured? Thanks! ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 6, 2022, 5:36am UTC](https://discuss.elastic.co/t/datanode-disk-full-despite-on-flood-stage-configuration/320468/2 "2022-12-06T05:36:57Z")
Welcome to our community! 😃 How much disk space do your nodes have? Cause yes, it should stop writing, but there's always a bit of wiggle room due to things like merges or reallocation that _may_ happen. So if your node has 5GB at 95% and your indices are 10's of GB in size then that _may_ explain it. But I am making a guess there without more info. ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [December 6, 2022, 7:24am UTC](https://discuss.elastic.co/t/datanode-disk-full-despite-on-flood-stage-configuration/320468/3 "2022-12-06T07:24:24Z")
It doesn't mean only ES fill your system. Might be others: paging, logs, processes.. as well. ---
**Author:** ![gidonshn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gidonshn/32/114231_2.png) [@gidonshn](https://discuss.elastic.co/u/gidonshn)\ **Post date:** [December 6, 2022, 7:31am UTC](https://discuss.elastic.co/t/datanode-disk-full-despite-on-flood-stage-configuration/320468/4 "2022-12-06T07:31:58Z")
Hi @warkolm Thanks for the welcome, and for the quick response. My "hot" nodes have 7TB of disk each (persisstent volume connected to the pod). So 5% of that is 350GB.. ---
**Author:** ![gidonshn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gidonshn/32/114231_2.png) [@gidonshn](https://discuss.elastic.co/u/gidonshn)\ **Post date:** [December 6, 2022, 7:40am UTC](https://discuss.elastic.co/t/datanode-disk-full-despite-on-flood-stage-configuration/320468/5 "2022-12-06T07:40:22Z")
Hi @Rios Thanks for the response. The FS that was filled up is used only for ES data since it's a persistent volume being mounted by the pod directory to the data directory. The mount point is: /usr/share/elasticsearch/data ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 3, 2023, 7:40am UTC](https://discuss.elastic.co/t/datanode-disk-full-despite-on-flood-stage-configuration/320468/6 "2023-01-03T07:40:35Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.