Index routing allocation does appear to move shards over to specified node

Elastic Stack Elasticsearch
1

I am trying to separate indexes so that they are assigned to specific nodes. I have been following this documentation in order to achieve that.

I have an Elasticsearch cluster with 3 nodes. Each node has been given a specific attribute node.attr.type, default, data and logging.

I can confirm this has been correctly set using

GET _nodes

...
        "node" : {
          "attr" : {
            "k8s_node_name" : "ip-10-0-1-215.us-east-2.compute.internal",
            "transform" : {
              "node" : "true"
            },
            "type" : "default",
            "xpack" : {
              "installed" : "true"
            },
....

I have an index (search_index_2) that i want to be allocated to this specific node (default). Currently the index is sharded over 2 different nodes. One of which is not of type default.

GET /_cat/shards

search_index_2                                                0 p STARTED  2931122 487.9mb 10.0.4.22  elasticsearch-staging-es-data-0
search_index_2                                                0 r STARTED  2931122 487.9mb 10.0.1.167 elasticsearch-staging-es-default-0

I have attempted to allocate the index to the specific node i want by setting its index.routing.allocation

PUT search_index_2/_settings
{
  "index.routing.allocation.require.type": "default"
}

GET search_index_2/_settings

{
  "search_index_2" : {
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "type" : "default"
            },
            "exclude" : {
              "type" : "data,logging"
            },
            "require" : {
              "type" : "default"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "search_index_2",
        "creation_date" : "1614600419741",
        "number_of_replicas" : "1",
        "uuid" : "vhUDcDbsQlW4T1gswq5xlQ",
        "version" : {
          "created" : "7090299"
        }
      }
    }
  }
}

However it is still sharded over both data and default when i only want it to be on default.

GET /_cat/shards

search_index_2                                                0 p STARTED  2931122 487.9mb 10.0.4.22  elasticsearch-staging-es-data-0
search_index_2                                                0 r STARTED  2931122 487.9mb 10.0.1.167 elasticsearch-staging-es-default-0
2

Can you run the following commands and share the results?

First run:

GET _nodes/type:data

And

GET _nodes/type:default

Then run an explain:

GET /_cluster/allocation/explain
{
  "index": "search_index_2",
  "shard": 0,
  "primary": true
}
3 
GET _nodes/type:data

{
  "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "cluster_name" : "elasticsearch-staging",
  "nodes" : {
    "5Sl5vunZSAu8JSzX-BNqjQ" : {
      "name" : "elasticsearch-staging-es-data-0",
      "transport_address" : "10.0.4.22:9300",
      "host" : "10.0.4.22",
      "ip" : "10.0.4.22",
      "version" : "7.16.2",
      "build_flavor" : "default",
      "build_type" : "docker",
      "build_hash" : "2b937c44140b6559905130a8650c64dbd0879cfb",
      "total_indexing_buffer" : 214748364,
      "roles" : [
        "data",
        "data_cold",
        "data_content",
        "data_frozen",
        "data_hot",
        "data_warm",
        "ingest",
        "master",
        "ml",
        "remote_cluster_client",
        "transform"
      ],
      "attributes" : {
        "k8s_node_name" : "ip-10-0-4-99.us-east-2.compute.internal",
        "ml.machine_memory" : "4294967296",
        "xpack.installed" : "true",
        "transform.node" : "true",
        "ml.max_open_jobs" : "512",
        "ml.max_jvm_size" : "2147483648",
        "type" : "data"
      },
....

GET _nodes/type:default

{
  "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "cluster_name" : "elasticsearch-staging",
  "nodes" : {
    "wjLRod1ESg-rwpyHRg5K2w" : {
      "name" : "elasticsearch-staging-es-default-0",
      "transport_address" : "10.0.1.167:9300",
      "host" : "10.0.1.167",
      "ip" : "10.0.1.167",
      "version" : "7.16.2",
      "build_flavor" : "default",
      "build_type" : "docker",
      "build_hash" : "2b937c44140b6559905130a8650c64dbd0879cfb",
      "total_indexing_buffer" : 214748364,
      "roles" : [
        "data",
        "data_cold",
        "data_content",
        "data_frozen",
        "data_hot",
        "data_warm",
        "ingest",
        "master",
        "ml",
        "remote_cluster_client",
        "transform"
      ],
      "attributes" : {
        "k8s_node_name" : "ip-10-0-1-215.us-east-2.compute.internal",
        "ml.machine_memory" : "4294967296",
        "xpack.installed" : "true",
        "transform.node" : "true",
        "ml.max_open_jobs" : "512",
        "ml.max_jvm_size" : "2147483648",
        "type" : "default"
      },
...

GET /_cluster/allocation/explain
{
  "index": "search_index_2",
  "shard": 0,
  "primary": true
}

{
  "index" : "search_index_2",
  "shard" : 0,
  "primary" : true,
  "current_state" : "started",
  "current_node" : {
    "id" : "5Sl5vunZSAu8JSzX-BNqjQ",
    "name" : "elasticsearch-staging-es-data-0",
    "transport_address" : "10.0.4.22:9300",
    "attributes" : {
      "k8s_node_name" : "ip-10-0-4-99.us-east-2.compute.internal",
      "ml.machine_memory" : "4294967296",
      "xpack.installed" : "true",
      "transform.node" : "true",
      "ml.max_open_jobs" : "512",
      "ml.max_jvm_size" : "2147483648",
      "type" : "data"
    }
  },
  "can_remain_on_current_node" : "no",
  "can_remain_decisions" : [
    {
      "decider" : "filter",
      "decision" : "NO",
      "explanation" : """node does not match index setting [index.routing.allocation.require] filters [type:"default"]"""
    }
  ],
  "can_move_to_other_node" : "no",
  "move_explanation" : "cannot move shard to another node, even though it is not allowed to remain on its current node",
  "node_allocation_decisions" : [
    {
      "node_id" : "I4JjWB9YQ7SaWbyV2leB3w",
      "node_name" : "elasticsearch-staging-es-logging-0",
      "transport_address" : "10.0.1.209:9300",
      "node_attributes" : {
        "k8s_node_name" : "ip-10-0-1-54.us-east-2.compute.internal",
        "ml.machine_memory" : "4294967296",
        "xpack.installed" : "true",
        "transform.node" : "true",
        "ml.max_open_jobs" : "512",
        "ml.max_jvm_size" : "2147483648",
        "type" : "logging"
      },
      "node_decision" : "no",
      "weight_ranking" : 1,
      "deciders" : [
        {
          "decider" : "filter",
          "decision" : "NO",
          "explanation" : """node does not match index setting [index.routing.allocation.require] filters [type:"default"]"""
        }
      ]
    },
    {
      "node_id" : "wjLRod1ESg-rwpyHRg5K2w",
      "node_name" : "elasticsearch-staging-es-default-0",
      "transport_address" : "10.0.1.167:9300",
      "node_attributes" : {
        "k8s_node_name" : "ip-10-0-1-215.us-east-2.compute.internal",
        "ml.machine_memory" : "4294967296",
        "xpack.installed" : "true",
        "transform.node" : "true",
        "ml.max_open_jobs" : "512",
        "ml.max_jvm_size" : "2147483648",
        "type" : "default"
      },
      "node_decision" : "no",
      "weight_ranking" : 2,
      "deciders" : [
        {
          "decider" : "same_shard",
          "decision" : "NO",
          "explanation" : "a copy of this shard is already allocated to this node [[search_index_2][0], node[wjLRod1ESg-rwpyHRg5K2w], [R], s[STARTED], a[id=iY556ql0TDqCE-pP0SwHAQ]]"
        },
        {
          "decider" : "awareness",
          "decision" : "NO",
          "explanation" : "there are [2] copies of this shard and [3] values for attribute [k8s_node_name] ([ip-10-0-1-215.us-east-2.compute.internal, ip-10-0-1-54.us-east-2.compute.internal, ip-10-0-4-99.us-east-2.compute.internal] from nodes in the cluster and no forced awareness) so there may be at most [1] copies of this shard allocated to nodes with each value, but (including this copy) there would be [2] copies allocated to nodes with [node.attr.k8s_node_name: ip-10-0-1-215.us-east-2.compute.internal]"
        }
      ]
    }
  ]
}

Had to cut of some information from the first 2 outputs due to the body limitations of this forum. But i think it is enough?

I can see the last command has shed more light on the issue, but not sure where to go from here...

4

From the explain comand it says that it cannot move it out from the node with the type data because the default node already has a copy of that shard, which is a replica.

How many nodes with the default type do you have?

If you just have one node with the default type, then you cannot have replicas for your index.

I think that if you remove the replica of your search_index_2 the cluster will move the shard out from the data node to the default node.

PUT /search_index_2/_settings
{
  "index" : {
    "number_of_replicas" : 0
  }
}
5
  • What is the future implication of setting number_of_replicas to 0?, ( i ahve never used/done anything with replicas before when it comes to indexes)

I spun up another node for that category of nodes so now have 1 -> 2 nodes.

And it looks like the search index has moved over now.

search_index_2                                                0 r STARTED     2931122 487.9mb 10.0.1.167 elasticsearch-staging-es-default-0
search_index_2                                                0 p STARTED     2931122 487.9mb 10.0.4.143 elasticsearch-staging-es-default-1

I wonder what happens if i scale back down... from 2 -> 1

6

It all depends on how many node do you have in your cluster.

Replicas allow you to have some redundancy, for example, if you have an index with 1 primary and 1 replica, then your index will have two shards, each one in a different node, if you lose one of the nodes, you still can query and write on that index.

If you have an index with 0 replicas, and the node where the shard of the index is stored has some problem, than you won't be able to query or write on that index until the node comes back.

Also, if your cluster has nodes with unallocated replicas, it will be in the yellow stage, if your cluster has unallocated primaries because one of the nodes went down, your cluster will be in the red state.

Your shards moved after you added another default node because now the cluster can allocate the primary shard on a default node and the replica on a different default node, replicas can't be allocated on the same node that already has the same primary shard.

7

Right makes a lot of sense.

I need to do this process for a few different indexes - (relocating them to the associated nodes). I'm not sure its suitable for me to configure the number_of_replicas for each. Mainly because it sounds good that there is a primary and replica. And i don't know what the default state of the indexes are/were. When an index is created does it always have 1 primary and 1 replica by default?

Could the process be to set the replicas to 0, and then set it back to 1?

  • An update on scaling back down, i can see that it has been in the process of migrating the data for a good 30 minutes now. Even though the index is only 900mb in size.

elasticsearch.elasticsearch.k8s.elastic.co/elasticsearch-staging green 4 7.16.2 MigratingData 4h47m

I'm going to assume something is failing here because of the index.routing.allocation rules that have been set that the index must only be on a default node and a primary and replica cannot exist on the same node.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.