Why does cluster.routing.allocation.exclude._ip only work as a transient, not persistent setting?

Just a sanity check... trying to remove a node by setting cluster.routing.allocation.exclude._ip does not seem to have any effect if it's a persistent setting. Tested with elasticesarch 7.17.13 on a 3-node cluster. When I changed it to transient setting it worked.
Is that expected?

No effect:

curl -XPUT http://192.168.1.50:9200/_cluster/settings --data-binary '{"persistent":{"cluster.routing.allocation.exclude._ip":"172.19.0.11" } }' -H 'Content-type: application/json'

This removed shards from that node:

curl -XPUT http://192.168.1.50:9200/_cluster/settings --data-binary '{"transient":{"cluster.routing.allocation.exclude._ip":"172.19.0.11" } }' -H 'Content-type: application/json'

A lot of docs like Cluster-level shard allocation and routing settings | Elasticsearch Guide [7.17] | Elastic and How to Remove a Node from an Elasticsearch Cluster use persistent so I feel like I'm missing something or this is a bug or problem with my testing?

Not directly expected, no, but the persistent setting will be ignored if the transient setting is present. If I had to guess, I would guess that you have both set. You can check to confirm this with GET _cluster/settings?flat_settings.

Thank you! Yes, I did have both persistent and transient settings.
Will reset and retest and followup if any problems.

Btw this is expected? If I only have two nodes and I exclude 1 of the nodes that has no effect? I guess it won't relocate shards if it has nowhere to put them?

% curl '192.168.1.50:9200/_cluster/settings?pretty&flat_settings'                              
{
  "persistent" : {
    "cluster.routing.allocation.exclude._ip" : "172.19.0.11"
  },
  "transient" : { }
}

% curl -fsS '192.168.1.50:9200/_cat/shards?pretty&v&h=index,shard,state,node,ip,prirep'|grep index
index                                                         shard state   node         ip          prirep
index_myindex                                      1     STARTED 9e26d679be62 172.19.0.14 p
index_myindex                                      1     STARTED 83b2b60f8596 172.19.0.11 r
index_myindex                                      4     STARTED 9e26d679be62 172.19.0.14 p
index_myindex                                      4     STARTED 83b2b60f8596 172.19.0.11 r
index_myindex                                      2     STARTED 9e26d679be62 172.19.0.14 p
index_myindex                                      2     STARTED 83b2b60f8596 172.19.0.11 r
index_myindex                                      3     STARTED 9e26d679be62 172.19.0.14 p
index_myindex                                      3     STARTED 83b2b60f8596 172.19.0.11 r
index_myindex                                      0     STARTED 9e26d679be62 172.19.0.14 p
index_myindex                                      0     STARTED 83b2b60f8596 172.19.0.11 r

% curl 'http://192.168.1.50:9200/_cat/nodes?pretty&v&h=name,id,ip,node.role,disk.total,disk.used,disk.avail&s=disk.used:desc'
name                id   ip          node.role disk.total disk.used disk.avail
elasticsearchleader jJS7 172.19.0.7  m            629.6gb   554.8gb     74.7gb
83b2b60f8596        v4lb 172.19.0.11 di           629.6gb   554.8gb     74.7gb
9e26d679be62        TScK 172.19.0.14 di           629.6gb   554.8gb     74.7gb

Yes that's right.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.