Index Allocation based on attribute not working

Elastic Stack Elasticsearch
#1

Hi!

Can someone pls explain to me why some shards dont get relocated to the correct node in my cluster? I want no shard on node-4 at the moment.

Infos:

GET _cat/nodeattrs

node-4 172.21.2.174 172.21.2.174 ml.machine_memory 17179869184
node-4 172.21.2.174 172.21.2.174 ml.max_open_jobs  20
node-4 172.21.2.174 172.21.2.174 xpack.installed   true
node-4 172.21.2.174 172.21.2.174 store_type        warm
node-4 172.21.2.174 172.21.2.174 transform.node    true
node-1 172.21.2.120 172.21.2.120 ml.machine_memory 33620926464
node-1 172.21.2.120 172.21.2.120 xpack.installed   true
node-1 172.21.2.120 172.21.2.120 transform.node    true
node-1 172.21.2.120 172.21.2.120 ml.max_open_jobs  20
node-1 172.21.2.120 172.21.2.120 store_type        hot
node-3 172.21.3.14  172.21.3.14  ml.machine_memory 17179869184
node-3 172.21.3.14  172.21.3.14  ml.max_open_jobs  20
node-3 172.21.3.14  172.21.3.14  xpack.installed   true
node-3 172.21.3.14  172.21.3.14  store_type        hot
node-3 172.21.3.14  172.21.3.14  transform.node    true
node-2 172.21.2.86  172.21.2.86  ml.machine_memory 33613492224
node-2 172.21.2.86  172.21.2.86  ml.max_open_jobs  20
node-2 172.21.2.86  172.21.2.86  xpack.installed   true
node-2 172.21.2.86  172.21.2.86  store_type        hot
node-2 172.21.2.86  172.21.2.86  transform.node    true

shards on node-4:

ppr-ecs-test-2020.09	0	p	STARTED	249	261.5kb	172.21.2.174	node-4

index settings:

{
  "ppr-ecs-test-2020.09" : {
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "store_type" : "hot"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "ppr-ecs-test-2020.09",
        "creation_date" : "1599654541084",
        "number_of_replicas" : "1",
        "uuid" : "CS-UYQWcSCKdi-NAFGTsdQ",
        "version" : {
          "created" : "7080099"
        }
      }
    }
  }
}
#2

Try the following

{
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "require" : {
              "store_type" : "hot"
            }
          }
        },
        "number_of_shards" : "1",
        "number_of_replicas" : "1"
        }
      }
    }
}

The difference between include and require:
Include means nodes may have that attribute
Require means nodes must have that attribute

#3

Can i overwrite the setting somehow? or delete the old one?

Because now i have this:

{
  "ppr-ecs-test-2020.09" : {
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "store_type" : "hot"
            },
            "require" : {
              "store_type" : "hot"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "ppr-ecs-test-2020.09",
        "creation_date" : "1599654541084",
        "number_of_replicas" : "1",
        "uuid" : "CS-UYQWcSCKdi-NAFGTsdQ",
        "version" : {
          "created" : "7080099"
        }
      }
    }
  }
}
#4 
"include" : {
              "store_type" : ""
            }

setting null value will essentially remove the setting

I would also recommend using index template for configuring this setting for all newly created indices where this setting is required.

#5

OK, now with:

{
  "ppr-ecs-test-2020.09" : {
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "store_type" : ""
            },
            "require" : {
              "store_type" : "hot"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "ppr-ecs-test-2020.09",
        "creation_date" : "1599654541084",
        "number_of_replicas" : "1",
        "uuid" : "CS-UYQWcSCKdi-NAFGTsdQ",
        "version" : {
          "created" : "7080099"
        }
      }
    }
  }
}

Its the same. Shard is on node-4 pretty much ignoring the setting.

Yeah for the new ones i do it via the template.

#6

Can you post the output of:
GET /_cat/recovery/ppr-ecs-test-2020.09

#7 
ppr-ecs-test-2020.09 0 576ms peer done 172.21.2.120 node-1 172.21.3.14  node-3 n/a n/a 0  0  0.0%   0  0      0      0.0%   0      0 0 100.0%
ppr-ecs-test-2020.09 0 2.5s  peer done 172.21.2.120 node-1 172.21.2.174 node-4 n/a n/a 16 16 100.0% 16 267821 267821 100.0% 267821 0 0 100.0%
#8

Can you share response of:
GET /allocation/explain?pretty

Can also try:
POST /_cluster/reroute?retry_failed=true

#9

GET /_cat/allocation/explain?pretty

EMPTY response

POST /_cluster/reroute?retry_failed=true

...
"ppr-ecs-test-2020.09" : {
          "shards" : {
            "0" : [
              {
                "state" : "STARTED",
                "primary" : false,
                "node" : "foRZv0ngQieXX3uwSXrZZg",
                "relocating_node" : null,
                "shard" : 0,
                "index" : "ppr-ecs-test-2020.09",
                "allocation_id" : {
                  "id" : "NP22yUdkTkmpCM4vvzrF7A"
                }
              },
              {
                "state" : "STARTED",
                "primary" : true,
                "node" : "zlcwsGyVSxOSE4c8I7MeVw",
                "relocating_node" : null,
                "shard" : 0,
                "index" : "ppr-ecs-test-2020.09",
                "allocation_id" : {
                  "id" : "On_WcGlVT9KJd-_tM06MYA"
                }
              }
            ]
          }
        }
...
#10

After the upgrade to 7.9.1 the shards got assigned on the correct node