Just a sanity check... trying to remove a node by setting cluster.routing.allocation.exclude._ip does not seem to have any effect if it's a persistent setting. Tested with elasticesarch 7.17.13 on a 3-node cluster. When I changed it to transient setting it worked.
Is that expected?
Not directly expected, no, but the persistent setting will be ignored if the transient setting is present. If I had to guess, I would guess that you have both set. You can check to confirm this with GET _cluster/settings?flat_settings.
Thank you! Yes, I did have both persistent and transient settings.
Will reset and retest and followup if any problems.
Btw this is expected? If I only have two nodes and I exclude 1 of the nodes that has no effect? I guess it won't relocate shards if it has nowhere to put them?
% curl '192.168.1.50:9200/_cluster/settings?pretty&flat_settings'
{
"persistent" : {
"cluster.routing.allocation.exclude._ip" : "172.19.0.11"
},
"transient" : { }
}
% curl -fsS '192.168.1.50:9200/_cat/shards?pretty&v&h=index,shard,state,node,ip,prirep'|grep index
index shard state node ip prirep
index_myindex 1 STARTED 9e26d679be62 172.19.0.14 p
index_myindex 1 STARTED 83b2b60f8596 172.19.0.11 r
index_myindex 4 STARTED 9e26d679be62 172.19.0.14 p
index_myindex 4 STARTED 83b2b60f8596 172.19.0.11 r
index_myindex 2 STARTED 9e26d679be62 172.19.0.14 p
index_myindex 2 STARTED 83b2b60f8596 172.19.0.11 r
index_myindex 3 STARTED 9e26d679be62 172.19.0.14 p
index_myindex 3 STARTED 83b2b60f8596 172.19.0.11 r
index_myindex 0 STARTED 9e26d679be62 172.19.0.14 p
index_myindex 0 STARTED 83b2b60f8596 172.19.0.11 r
% curl 'http://192.168.1.50:9200/_cat/nodes?pretty&v&h=name,id,ip,node.role,disk.total,disk.used,disk.avail&s=disk.used:desc'
name id ip node.role disk.total disk.used disk.avail
elasticsearchleader jJS7 172.19.0.7 m 629.6gb 554.8gb 74.7gb
83b2b60f8596 v4lb 172.19.0.11 di 629.6gb 554.8gb 74.7gb
9e26d679be62 TScK 172.19.0.14 di 629.6gb 554.8gb 74.7gb
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.