Why does cluster.routing.allocation.exclude._ip only work as a transient, not persistent setting?

Elastic Stack Elasticsearch
1

Just a sanity check... trying to remove a node by setting cluster.routing.allocation.exclude._ip does not seem to have any effect if it's a persistent setting. Tested with elasticesarch 7.17.13 on a 3-node cluster. When I changed it to transient setting it worked.
Is that expected?

No effect:

curl -XPUT http://192.168.1.50:9200/_cluster/settings --data-binary '{"persistent":{"cluster.routing.allocation.exclude._ip":"172.19.0.11" } }' -H 'Content-type: application/json'

This removed shards from that node:

curl -XPUT http://192.168.1.50:9200/_cluster/settings --data-binary '{"transient":{"cluster.routing.allocation.exclude._ip":"172.19.0.11" } }' -H 'Content-type: application/json'

A lot of docs like Cluster-level shard allocation and routing settings | Elasticsearch Guide [7.17] | Elastic and How to Remove a Node from an Elasticsearch Cluster use persistent so I feel like I'm missing something or this is a bug or problem with my testing?

2

Not directly expected, no, but the persistent setting will be ignored if the transient setting is present. If I had to guess, I would guess that you have both set. You can check to confirm this with GET _cluster/settings?flat_settings.

3

Thank you! Yes, I did have both persistent and transient settings.
Will reset and retest and followup if any problems.

Btw this is expected? If I only have two nodes and I exclude 1 of the nodes that has no effect? I guess it won't relocate shards if it has nowhere to put them?

% curl '192.168.1.50:9200/_cluster/settings?pretty&flat_settings'                              
{
  "persistent" : {
    "cluster.routing.allocation.exclude._ip" : "172.19.0.11"
  },
  "transient" : { }
}

% curl -fsS '192.168.1.50:9200/_cat/shards?pretty&v&h=index,shard,state,node,ip,prirep'|grep index
index                                                         shard state   node         ip          prirep
index_myindex                                      1     STARTED 9e26d679be62 172.19.0.14 p
index_myindex                                      1     STARTED 83b2b60f8596 172.19.0.11 r
index_myindex                                      4     STARTED 9e26d679be62 172.19.0.14 p
index_myindex                                      4     STARTED 83b2b60f8596 172.19.0.11 r
index_myindex                                      2     STARTED 9e26d679be62 172.19.0.14 p
index_myindex                                      2     STARTED 83b2b60f8596 172.19.0.11 r
index_myindex                                      3     STARTED 9e26d679be62 172.19.0.14 p
index_myindex                                      3     STARTED 83b2b60f8596 172.19.0.11 r
index_myindex                                      0     STARTED 9e26d679be62 172.19.0.14 p
index_myindex                                      0     STARTED 83b2b60f8596 172.19.0.11 r

% curl 'http://192.168.1.50:9200/_cat/nodes?pretty&v&h=name,id,ip,node.role,disk.total,disk.used,disk.avail&s=disk.used:desc'
name                id   ip          node.role disk.total disk.used disk.avail
elasticsearchleader jJS7 172.19.0.7  m            629.6gb   554.8gb     74.7gb
83b2b60f8596        v4lb 172.19.0.11 di           629.6gb   554.8gb     74.7gb
9e26d679be62        TScK 172.19.0.14 di           629.6gb   554.8gb     74.7gb
4

Yes that's right.