Disabling shard allocation not working as expected

Brent_Plummer · August 18, 2023, 8:49pm

One of my 6 Elasticsearch clusters at work is not respecting when i try to disable shard allocation prior to pull a node/server from the cluster.

When I run the curl command below I get the "true" response you would expect, also shown below. However, after I pull a node from the cluster it will immediately start to rebalance shards to make up for the lost node. I've confirmed that my command does make it to each node when looking at the _cluster/settings?include_defaults=true call. I've attached those setting as well.

Please let me know if you happen to see any configuration that may be causing the problem.

curl --insecure -u admin:<PASSWORD> -X PUT "https://contoso:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.routing.allocation.enable": "none"
  }
}
'

{
  "acknowledged" : true,
  "persistent" : { },
  "transient" : { }
}

leandrojmp · August 18, 2023, 8:59pm

Can you provide a little more context of what you want to do?

This request will clear the setting cluster.routing.allocation.enable and then it will use the default configuration, which is the all option.

If you want to remove a node from the cluster, you should exclude the node from shard allocation with the following request as explained in the documentation.

PUT _cluster/settings
{
  "persistent" : {
    "cluster.routing.allocation.exclude._ip" : "node-ip"
  }
}

This will make elasticsearch to start emptying the node and moving the shards to other nodes.

Brent_Plummer · August 18, 2023, 9:19pm

Thanks for the quick reply! My apologies. I pasted the wrong text above and have corrected it. The command should change the setting to "none". I understand that null will set it back to the default.

Basically, I'm just following the suggested procedures to do a rolling upgrade the ES software in a cluster. The "Disable shard allocation" step in the link below.

Note, I have tried "primaries" and "new_primaries" but same thing occurs. Cluster will try to rebalance as soon as a node leaves.

I have not tried to use the exclude.__ip option but really don't think that should be necessary per the rolling upgrade instructions. As i noted, I have about 5 other clusters where I have no problems with these setting changes being enforced.

leandrojmp · August 18, 2023, 10:18pm

You are right, you should not use exclude if you want to do a rolling upgrade, it was not clear on your first post so I understood that you wanted to remove a node from the cluster.

You should set the alloction to primaries as the following request:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}

But your issue is probably here:

You have a cluster.routing.allocation.enable set to all as a transient setting and the value for the transient will take precedence over the value of the same persistent setting, so it does not matter what you set in the persistent.cluster.routing.allocation.enable, it will be ignored.

You should first remove this transient setting, and then change the persistent setting.

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.enable": null
  }
}

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}

Brent_Plummer · August 18, 2023, 11:20pm

Oh man, yeah, it was right there! I kinda of feel embarrassed as I was comparing these settings and must have looked right past the transient setting. It would have been a lot easier to notice if I would have just thrown on &pretty=true to the _cluster/settings?include_defaults=true call.

Of course, makes a lot of sense now. Thanks so much for taking the time to read over and explain things clearly. Sometimes you just need a second set of eyes and some more experience. Appreciate it.

system · September 15, 2023, 11:20pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.