(bump) shards constantly relocating

[raymondmintz11 (raymond)]

hi
sorry if there is another way to bumb expired topics/comments
Was there a solution to this dialogue ?

Elasticsearch Constantly Reallocating Shards

ES Version 5.6.2 For the past few weeks, every time I check ES Monitoring, twenty shards are being reallocated: [image] This is defined as the max in my cluster settings. I have it set higher than default as many of the machines I'm running on are random servers and not the most reliable. Hence if a node fails I want the cluster to dynamically reallocate shards. However no nodes have gone down in the past few weeks... GET _cluster/settings { "persistent": { "cluster": { "routing…

Thanks!

(sorry for the irrelevant tag, the preset options arent applicable)

[leandrojmp (Leandro Pereira)]

Which version are you having this issue? I suggest you open a new topic or search for one similar for a more recently version.

The version linked in the post is pretty old.

[raymondmintz11 (raymond)]

OK , ill look around thanks

"version" : {
    "number" : "8.6.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "2d58d0f136141f03239816a4e360a8d17b6d8f29",
    "build_date" : "2023-02-13T09:35:20.314882762Z",
    "build_snapshot" : false,
    "lucene_version" : "9.4.2",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"

[leandrojmp (Leandro Pereira)]

Did you change the allocation settings?

More specifically these two:

cluster.routing.allocation.cluster_concurrent_rebalance
cluster.routing.allocation.node_concurrent_recoveries

There is an open issue about it, you can check it here.

[raymondmintz11 (raymond)]

hi again,
so i found this in my search, looks like the scenario is different though: no upgrade or high watermark issues in my use case:

Cluster shards unbalanced and keep moving shards around after upgrade to 8.8.1

Hello @DavidTurner, There was another issue, a couple of the nodes were hitting the watermarks and others were close to it, so probably any shard movement could trigger another watermark that would then trigger another shard movement. I removed some old data to make sure that no node in the warm tier would hit any watermark, and after sometime there are no more shard movements. $ cat desired-balance.json | grep "node_is_desired" | grep -v relocating | grep false | wc -l 0

Is there a post somewhere you can reference ?

[raymondmintz11 (raymond)]

no changes there. should be default

[leandrojmp (Leandro Pereira)]

Use GET _cluster/settings to check it.

This is still an open issue, the post you shared, that I opened when I had a similar issue has some discussion about it and the github issue linked has some information as well.

[raymondmintz11 (raymond)]

yesir, did that,
looks all vanilla

[Thu Mar 07 15:37:26] root@server.ebo:~# cat /EdgeCast/soc/utils/es/cmd/settings.sh 
#!/bin/bash

curl -s http://localhost:9200/_cluster/settings | jq . 
[Thu Mar 07 15:37:33] root@server.ebo:~# settings.sh 
{
  "persistent": {},
  "transient": {
    "cluster": {
      "routing": {
        "allocation": {
          "exclude": {
            "_name": "stuff and stuff"
          }
        }
      }
    }
  }
}
[Thu Mar 07 15:37:42] root@server.ebo:~#