Can't update disk space watermarks via API


(PaWeLL) #1

Hello there.
I'm trying to update thresholds of disk space in my ES cluster but can't do this due to error.

[admin@client1 ~]$ curl -X PUT "10.xxx.xxx.xxx:9200/_cluster/settings?flat_settings=true" -H 'Content- 
Type: application/json' -d'
{
"transient" : {
    "cluster.routing.allocation.disk.watermark.low" : "93%"
}, "persistent" : {
    "cluster.routing.allocation.disk.watermark.low" : "93%"
}
}
'
{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[Master1][10.xxx.xxx.xxx:9300][cluster:admin/settings/update]"}],"type":"illegal_argument_exception","reason":"low disk watermark [93%] more than high disk watermark [90%]"},"status":400}

The problem is that high disk watermark is already set to 95%. I checked elasticsearch.yml, persistent settings and transient settings. Everywhere high watermark is 95%.
The same problem occurs when i try to increase high watermark. It says that flood watermark is 95% but it's 98% in fact.

Where does these settings come from?


(PaWeLL) #2

Question is still not solved :frowning:


(David Pilato) #3

Did you try to first increase cluster.routing.allocation.disk.watermark.high to 95% for example?

Note that you can disable watermark with cluster.routing.allocation.disk.threshold_enabled: false. But this is not recommended for production clusters.

You should really have more disk space. You can do that by adding more disk, adding more nodes or cleaning non needed data.

My 2 cents.


(PaWeLL) #4

When I try to increase disk.watermark.high it says that flood_stage watermark (95%) is lower than new value of disk.watermark.high, but this is not true too.
I already set flood_stage watermark to 98%.


(David Pilato) #5

Did you try the second option I gave?


(PaWeLL) #6

Are you talking about disabling watermarks? If yes, I didn't tried it.


(David Pilato) #7

Yes I do.