# What's the disk watermark actually applied while specified in both node configuration file and cluster settings? **URL:** \ **Category:** Elasticsearch\ **Created:** [February 23, 2022, 3:56am UTC](https://discuss.elastic.co/t/whats-the-disk-watermark-actually-applied-while-specified-in-both-node-configuration-file-and-cluster-settings/297966 "2022-02-23T03:56:38Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![Hao\_Yellow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hao_yellow/32/102182_2.png) [@Hao\_Yellow](https://discuss.elastic.co/u/Hao_Yellow)\ **Post date:** [February 23, 2022, 3:56am UTC](https://discuss.elastic.co/t/whats-the-disk-watermark-actually-applied-while-specified-in-both-node-configuration-file-and-cluster-settings/297966/1 "2022-02-23T03:56:38Z")
Hello, I have a cluster of several nodes, where each node has the following configuration about the disk watermark: ```auto cluster.routing.allocation.disk.watermark.low=60gb cluster.routing.allocation.disk.watermark.high=50gb cluster.routing.allocation.disk.watermark.flood_stage=30gb ``` If I change the cluster watermark in Kibana Dev Tools using the following command, what will be the finally applied watermark values? ```auto PUT _cluster/settings { "persistent": { "cluster.routing.allocation.disk.watermark.low": "100gb", "cluster.routing.allocation.disk.watermark.high": "50gb", "cluster.routing.allocation.disk.watermark.flood_stage": "10gb" } } ``` Besides, what would happen if I set different watermark values in each node's configuration file? Thanks for you help! ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 23, 2022, 3:58am UTC](https://discuss.elastic.co/t/whats-the-disk-watermark-actually-applied-while-specified-in-both-node-configuration-file-and-cluster-settings/297966/2 "2022-02-23T03:58:59Z")
Welcome to our community! 😃 > [@Hao\_Yellow](#): > > If I change the cluster watermark in Kibana Dev Tools using the following command, what will be the finally applied watermark values? It will be the ones you posted, as they are cluster level. ---
**Author:** ![Hao\_Yellow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hao_yellow/32/102182_2.png) [@Hao\_Yellow](https://discuss.elastic.co/u/Hao_Yellow)\ **Post date:** [February 23, 2022, 6:20am UTC](https://discuss.elastic.co/t/whats-the-disk-watermark-actually-applied-while-specified-in-both-node-configuration-file-and-cluster-settings/297966/3 "2022-02-23T06:20:05Z")
Thanks! So if I don't post cluster settings, how does the cluster determine the watermark? I think it should follow the configuration file for each node. For example, In a cluster, here's the configuration for each node: ```auto cluster.routing.allocation.disk.watermark.low=60gb cluster.routing.allocation.disk.watermark.high=50gb cluster.routing.allocation.disk.watermark.flood_stage=30gb ``` Since each node has its own configuration file, one immediate question and curiosity is: **What would happen if I set different watermark settings for each node in the configuration?** (When I don't post any cluster settings on that) I haven't tried it myself, but I don't find any documentation about this in Elasticsearch Guide. Thanks again. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [February 23, 2022, 7:36am UTC](https://discuss.elastic.co/t/whats-the-disk-watermark-actually-applied-while-specified-in-both-node-configuration-file-and-cluster-settings/297966/4 "2022-02-23T07:36:04Z")
> [@Hao\_Yellow](#): > > What would happen if I set different watermark settings for each node in the configuration? Shard allocation decisions, including those involving disk watermarks, are taken on the elected master, so it's the settings on the master that matter. ---
**Author:** ![casterQ](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/casterq/32/93257_2.png) [@casterQ](https://discuss.elastic.co/u/casterQ)\ **Post date:** [February 23, 2022, 9:56am UTC](https://discuss.elastic.co/t/whats-the-disk-watermark-actually-applied-while-specified-in-both-node-configuration-file-and-cluster-settings/297966/5 "2022-02-23T09:56:10Z")
first，the priority of settings method is as follows: **transient -\> persistent -\> yml** restart the cluster transient settings will invalid， transient and persistent settings are all Cluster level。 second，If you configure different nodes with different yml settings，each node executes its own policy， ---
**Author:** ![Hao\_Yellow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hao_yellow/32/102182_2.png) [@Hao\_Yellow](https://discuss.elastic.co/u/Hao_Yellow)\ **Post date:** [February 23, 2022, 10:19am UTC](https://discuss.elastic.co/t/whats-the-disk-watermark-actually-applied-while-specified-in-both-node-configuration-file-and-cluster-settings/297966/6 "2022-02-23T10:19:36Z")
Thanks! I'm not sure I understand correctly what means "Cluster level" here? > [@casterQ](#): > > transient and persistent settings are all Cluster level。 In my understanding, when I have set **persistent** watermarks, the watermarks in yaml configuration of each node will be ignored, is that true? ---
**Author:** ![casterQ](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/casterq/32/93257_2.png) [@casterQ](https://discuss.elastic.co/u/casterQ)\ **Post date:** [February 23, 2022, 10:23am UTC](https://discuss.elastic.co/t/whats-the-disk-watermark-actually-applied-while-specified-in-both-node-configuration-file-and-cluster-settings/297966/7 "2022-02-23T10:23:22Z")
> [@casterQ](#): > > transient -\> persistent -\> yml yes, this is the conf priority : **transient -\> persistent -\> yml** ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [February 23, 2022, 10:50am UTC](https://discuss.elastic.co/t/whats-the-disk-watermark-actually-applied-while-specified-in-both-node-configuration-file-and-cluster-settings/297966/8 "2022-02-23T10:50:24Z")
Perhaps worth linking to the docs about this since they have some more detail: > **[Cluster update settings API | Elasticsearch Guide \[8.0\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-update-settings.html)** ---
**Author:** ![Hao\_Yellow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hao_yellow/32/102182_2.png) [@Hao\_Yellow](https://discuss.elastic.co/u/Hao_Yellow)\ **Post date:** [February 23, 2022, 10:51am UTC](https://discuss.elastic.co/t/whats-the-disk-watermark-actually-applied-while-specified-in-both-node-configuration-file-and-cluster-settings/297966/9 "2022-02-23T10:51:29Z")
One more thing? About this: > [@casterQ](#): > > second，If you configure different nodes with different yml settings，each node executes its own policy， Another friend's reply says differently, where in this case "each node will follow master node's settings". ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [February 23, 2022, 10:53am UTC](https://discuss.elastic.co/t/whats-the-disk-watermark-actually-applied-while-specified-in-both-node-configuration-file-and-cluster-settings/297966/10 "2022-02-23T10:53:33Z")
> [@Hao\_Yellow](#): > > Another friend's reply says differently, where in this case "each node will follow master node's settings". Some settings take effect at the node level so you can set them differently on each node. But the disk watermarks aren't like that because they affect shard allocation decisions which are only computed on the master, so it's the master's setting which matters. ---
**Author:** ![Hao\_Yellow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hao_yellow/32/102182_2.png) [@Hao\_Yellow](https://discuss.elastic.co/u/Hao_Yellow)\ **Post date:** [February 23, 2022, 11:16am UTC](https://discuss.elastic.co/t/whats-the-disk-watermark-actually-applied-while-specified-in-both-node-configuration-file-and-cluster-settings/297966/11 "2022-02-23T11:16:44Z")
Thanks for pointing out the reference and the clear answer, David! So, may I ask one more question here? **Background:** Let's say we have disk flood stage watermark set to "30 GB" — then if one node has reaches this watermark (has less than 30 GB disk available), the indices which have shards in the node will be marked as **read\_only\_allow\_delete** (is that right?). And the documentation says "_When disk usage on the affected node drops below the high watermark, Elasticsearch automatically removes the write block._". **My Issue:** Now, recently I've encountered a problem where I see some indices have been automatically set this read\_only\_allow\_delete flag. I checked the cluster allocation, and the available disk space is more than 50 GB, which is far greater than the 30 GB flood stage, and in this case, the 50 GB high watermark. But still, the flag isn't automatically removed from the indices. So I had to remove the flags myself. Do you have thoughts on this problem? Thanks! ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [February 23, 2022, 11:22am UTC](https://discuss.elastic.co/t/whats-the-disk-watermark-actually-applied-while-specified-in-both-node-configuration-file-and-cluster-settings/297966/12 "2022-02-23T11:22:17Z")
Elasticsearch emits logs that describe the decisions it's making in relation to disk watermarks. You'll need to look at them, or share them here (in full) if you would like help interpreting them. ---
**Author:** ![Hao\_Yellow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hao_yellow/32/102182_2.png) [@Hao\_Yellow](https://discuss.elastic.co/u/Hao_Yellow)\ **Post date:** [February 23, 2022, 2:30pm UTC](https://discuss.elastic.co/t/whats-the-disk-watermark-actually-applied-while-specified-in-both-node-configuration-file-and-cluster-settings/297966/13 "2022-02-23T14:30:49Z")
Thank you, David. Yeah, I checked the logs last time. Actually the cluster I'm talking about is deployed as docker containers. When I checked the logs last time, I used the following command on the node server where there was least available disk space of all nodes in the cluster (because I thought that was the problem node!): ```auto docker container logs es_container_id --since '2022-02-22' ``` But without any valuable content about the "read\_only\_allow\_delete" or about "watermark"! **Now that you mentioned the logs, I am wondering:** When I want to see Elasticsearch logs about this _read\_only\_allow\_delete_ issue (why and when Elasticsearch adds the flag), which node server in the cluster should I log in to find the logs? The master node, or the one with least disk space available? Or are there other better ways to view the logs to debug this problem? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [February 23, 2022, 3:21pm UTC](https://discuss.elastic.co/t/whats-the-disk-watermark-actually-applied-while-specified-in-both-node-configuration-file-and-cluster-settings/297966/14 "2022-02-23T15:21:44Z")
I forget, but it might be the master. I'd normally just look at them all. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 23, 2022, 3:21pm UTC](https://discuss.elastic.co/t/whats-the-disk-watermark-actually-applied-while-specified-in-both-node-configuration-file-and-cluster-settings/297966/15 "2022-03-23T15:21:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.