Can't update cluster watermarks

Elastic Stack Elasticsearch
1

Hi,

I got some exceptions in my logfiles:

Caused by: java.lang.IllegalArgumentException: unable to consistently parse     [cluster.routing.allocation.disk.watermark.low=20gb], [cluster.routing.allocation.disk.watermark.high=10gb], and [cluster.routing.allocation.disk.watermark.flood_stage=95%] as percentage or bytes
        at org.elasticsearch.cluster.routing.allocation.DiskThresholdSettings.doValidate(DiskThresholdSettings.java:170) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.routing.allocation.DiskThresholdSettings.access$000(DiskThresholdSettings.java:38) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.routing.allocation.DiskThresholdSettings$LowDiskWatermarkValidator.validate(DiskThresholdSettings.java:101) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.routing.allocation.DiskThresholdSettings$LowDiskWatermarkValidator.validate(DiskThresholdSettings.java:95) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.settings.Setting.get(Setting.java:362) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.settings.Setting.get(Setting.java:343) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.settings.Setting$Updater.getValue(Setting.java:934) ~[elasticsearch-6.3.0.jar:6.3.0]
        ... 11 more
Caused by: org.elasticsearch.ElasticsearchParseException: failed to parse setting [cluster.routing.allocation.disk.watermark.flood_stage] with value [95%] as a size in bytes: unit is missing or unrecognized
        at org.elasticsearch.common.unit.ByteSizeValue.parseBytesSizeValue(ByteSizeValue.java:225) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.unit.ByteSizeValue.parseBytesSizeValue(ByteSizeValue.java:185) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.routing.allocation.DiskThresholdSettings.thresholdBytesFromWatermark(DiskThresholdSettings.java:343) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.routing.allocation.DiskThresholdSettings.doValidateAsBytes(DiskThresholdSettings.java:194) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.routing.allocation.DiskThresholdSettings.doValidate(DiskThresholdSettings.java:159) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.routing.allocation.DiskThresholdSettings.access$000(DiskThresholdSettings.java:38) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.routing.allocation.DiskThresholdSettings$LowDiskWatermarkValidator.validate(DiskThresholdSettings.java:101) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.routing.allocation.DiskThresholdSettings$LowDiskWatermarkValidator.validate(DiskThresholdSettings.java:95) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.settings.Setting.get(Setting.java:362) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.settings.Setting.get(Setting.java:343) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.settings.Setting$Updater.getValue(Setting.java:934) ~[elasticsearch-6.3.0.jar:6.3.0]

If I try to update the watermarks, I get an error message:

curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
> {
>   "transient": {
>     "cluster.routing.allocation.disk.watermark.low": "20gb",
>     "cluster.routing.allocation.disk.watermark.high": "10gb"
>   }
> }
> '
{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[hostihost][xxx.xxx.xxx.xxx:9300][cluster:admin/settings/update]"}],"type":"illegal_argument_exception","reason":"unable to consistently parse [cluster.routing.allocation.disk.watermark.low=20gb], [cluster.routing.allocation.disk.watermark.high=10gb], and [cluster.routing.allocation.disk.watermark.flood_stage=95%] as percentage or bytes","caused_by":{"type":"parse_exception","reason":"failed to parse setting [cluster.routing.allocation.disk.watermark.flood_stage] with value [95%] as a size in bytes: unit is missing or unrecognized"}},"status":400}

Any help is appreciated.

2

There are three watermark settings, which all need to be set and correct in relation to each other. Ir seems the flood_stage setting, and the default setting does not align with the updated settings you are providing.

3

Thank you for the quick response. I put the flood_stage in my curl, but that doesn't help:

curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
>  {
>    "transient": {
>      "cluster.routing.allocation.disk.watermark.low": "20gb",
>      "cluster.routing.allocation.disk.watermark.high": "10gb",
>      "cluster.routing.allocation.disk.watermark.flood_stage": "5gb"
>   }
> }
> '
{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[hostihost][xxx.xxx.xxx.xxx:9300][cluster:admin/settings/update]"}],"type":"illegal_argument_exception","reason":"unknown setting [archived.indices.recovery.concurrent_streams] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"},"status":400}
4

It seems like it is complaining about archived.indices.recovery.concurrent_streams. Do you have this setting somewhere? Which version of Elasticsearch are you running?

5

I checked my elasticsearch.yml and I can't find any watermark configs in there.
I have 6.3 installed right now.

6

What do you get through the get cluster settings API?

7 
curl -XGET "localhost:9200/_cluster/settings?pretty=true"
{
  "persistent" : {
    "archived" : {
      "indices" : {
        "recovery" : {
          "concurrent_streams" : "5"
        }
      }
    },
    "cluster" : {
      "routing" : {
        "allocation" : {
          "node_concurrent_recoveries" : "5",
          "enable" : "none"
        }
      }
    },
    "indices" : {
      "recovery" : {
        "max_bytes_per_sec" : "200mb"
      }
    }
  },
  "transient" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "disk" : {
            "watermark" : {
              "low" : "20gb",
              "high" : "10gb"
            }
          },
          "enable" : "all"
        }
      },
      "info" : {
        "update" : {
          "interval" : "1m"
        }
      }
    }
  }
}
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.