Index Allocation based on attribute not working

Elastic Stack Elasticsearch
1

Hi!

Can someone pls explain to me why some shards dont get relocated to the correct node in my cluster? I want no shard on node-4 at the moment.

Infos:

GET _cat/nodeattrs

node-4 172.21.2.174 172.21.2.174 ml.machine_memory 17179869184
node-4 172.21.2.174 172.21.2.174 ml.max_open_jobs  20
node-4 172.21.2.174 172.21.2.174 xpack.installed   true
node-4 172.21.2.174 172.21.2.174 store_type        warm
node-4 172.21.2.174 172.21.2.174 transform.node    true
node-1 172.21.2.120 172.21.2.120 ml.machine_memory 33620926464
node-1 172.21.2.120 172.21.2.120 xpack.installed   true
node-1 172.21.2.120 172.21.2.120 transform.node    true
node-1 172.21.2.120 172.21.2.120 ml.max_open_jobs  20
node-1 172.21.2.120 172.21.2.120 store_type        hot
node-3 172.21.3.14  172.21.3.14  ml.machine_memory 17179869184
node-3 172.21.3.14  172.21.3.14  ml.max_open_jobs  20
node-3 172.21.3.14  172.21.3.14  xpack.installed   true
node-3 172.21.3.14  172.21.3.14  store_type        hot
node-3 172.21.3.14  172.21.3.14  transform.node    true
node-2 172.21.2.86  172.21.2.86  ml.machine_memory 33613492224
node-2 172.21.2.86  172.21.2.86  ml.max_open_jobs  20
node-2 172.21.2.86  172.21.2.86  xpack.installed   true
node-2 172.21.2.86  172.21.2.86  store_type        hot
node-2 172.21.2.86  172.21.2.86  transform.node    true

shards on node-4:

ppr-ecs-test-2020.09	0	p	STARTED	249	261.5kb	172.21.2.174	node-4

index settings:

{
  "ppr-ecs-test-2020.09" : {
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "store_type" : "hot"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "ppr-ecs-test-2020.09",
        "creation_date" : "1599654541084",
        "number_of_replicas" : "1",
        "uuid" : "CS-UYQWcSCKdi-NAFGTsdQ",
        "version" : {
          "created" : "7080099"
        }
      }
    }
  }
}
2

Try the following

{
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "require" : {
              "store_type" : "hot"
            }
          }
        },
        "number_of_shards" : "1",
        "number_of_replicas" : "1"
        }
      }
    }
}

The difference between include and require:
Include means nodes may have that attribute
Require means nodes must have that attribute

3

Can i overwrite the setting somehow? or delete the old one?

Because now i have this:

{
  "ppr-ecs-test-2020.09" : {
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "store_type" : "hot"
            },
            "require" : {
              "store_type" : "hot"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "ppr-ecs-test-2020.09",
        "creation_date" : "1599654541084",
        "number_of_replicas" : "1",
        "uuid" : "CS-UYQWcSCKdi-NAFGTsdQ",
        "version" : {
          "created" : "7080099"
        }
      }
    }
  }
}
4 
"include" : {
              "store_type" : ""
            }

setting null value will essentially remove the setting

I would also recommend using index template for configuring this setting for all newly created indices where this setting is required.

5

OK, now with:

{
  "ppr-ecs-test-2020.09" : {
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "store_type" : ""
            },
            "require" : {
              "store_type" : "hot"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "ppr-ecs-test-2020.09",
        "creation_date" : "1599654541084",
        "number_of_replicas" : "1",
        "uuid" : "CS-UYQWcSCKdi-NAFGTsdQ",
        "version" : {
          "created" : "7080099"
        }
      }
    }
  }
}

Its the same. Shard is on node-4 pretty much ignoring the setting.

Yeah for the new ones i do it via the template.

6

Can you post the output of:
GET /_cat/recovery/ppr-ecs-test-2020.09

7 
ppr-ecs-test-2020.09 0 576ms peer done 172.21.2.120 node-1 172.21.3.14  node-3 n/a n/a 0  0  0.0%   0  0      0      0.0%   0      0 0 100.0%
ppr-ecs-test-2020.09 0 2.5s  peer done 172.21.2.120 node-1 172.21.2.174 node-4 n/a n/a 16 16 100.0% 16 267821 267821 100.0% 267821 0 0 100.0%
8

Can you share response of:
GET /allocation/explain?pretty

Can also try:
POST /_cluster/reroute?retry_failed=true

9

GET /_cat/allocation/explain?pretty

EMPTY response

POST /_cluster/reroute?retry_failed=true

...
"ppr-ecs-test-2020.09" : {
          "shards" : {
            "0" : [
              {
                "state" : "STARTED",
                "primary" : false,
                "node" : "foRZv0ngQieXX3uwSXrZZg",
                "relocating_node" : null,
                "shard" : 0,
                "index" : "ppr-ecs-test-2020.09",
                "allocation_id" : {
                  "id" : "NP22yUdkTkmpCM4vvzrF7A"
                }
              },
              {
                "state" : "STARTED",
                "primary" : true,
                "node" : "zlcwsGyVSxOSE4c8I7MeVw",
                "relocating_node" : null,
                "shard" : 0,
                "index" : "ppr-ecs-test-2020.09",
                "allocation_id" : {
                  "id" : "On_WcGlVT9KJd-_tM06MYA"
                }
              }
            ]
          }
        }
...
10

After the upgrade to 7.9.1 the shards got assigned on the correct node

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.