Manually move replica shard to another node

Elastic Stack Elasticsearch
1

I have an unassigned replica shard that I need to assign to a node but the only node available that has "room" already has the same primary shard assigned to it. My setup:
4 node cluster with 1 replica. I only have an issue with one index the rest are allocated properly.

Node1  Node2  Node3  Node4  UNASSIGNED
P1 R2  R0 P2  P3     P0 R1	R3

I would like to move replica 1 to Node3 and then run the allocate_replica command to resolve this issue(if that's the best approach)?

2

You want to force allocation to Node3? How much space does it have?

3

@warkolm
my apologies for any confusion...what I mean by "room" is, its the only node that is missing a shard. the unassigned shard will not allocate to other shards because currently there are 2 shards already assigned and I am only allowing 2 shards per node. Here is the output from allocation explained command:

{
  "index" : "sessions2-200920",
  "shard" : 3,
  "primary" : false,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "REPLICA_ADDED",
    "at" : "2020-09-21T06:11:14.548Z",
    "last_allocation_status" : "no_attempt"
  },
  "can_allocate" : "no",
  "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
  "node_allocation_decisions" : [
    {
      "node_id" : "9xxx99xxxbb8S1yDg6xxx",
      "node_name" : "node2",
      "transport_address" : "xxx.xxx.xx.xx:9300",
      "node_attributes" : {
        "ml.machine_memory" : "67190153216",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true",
        "transform.node" : "true"
      },
      "node_decision" : "no",
      "deciders" : [
        {
          "decider" : "shards_limit",
          "decision" : "NO",
          "explanation" : "too many shards [2] allocated to this node for index [sessions2-200920], index setting [index.routing.allocation.total_shards_per_node=2]"
        }
      ]
    },
    {
      "node_id" : "BVtxxxdvSxxxqXxeFLWxxx_lA",
      "node_name" : "node1",
      "transport_address" : "xxx.xxx.xx.xx:9300",
      "node_attributes" : {
        "ml.machine_memory" : "67190153216",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true",
        "transform.node" : "true"
      },
      "node_decision" : "no",
      "deciders" : [
        {
          "decider" : "shards_limit",
          "decision" : "NO",
          "explanation" : "too many shards [2] allocated to this node for index [sessions2-200920], index setting [index.routing.allocation.total_shards_per_node=2]"
        }
      ]
    },
    {
      "node_id" : "MxxxiYkaQMSpzgxxx_0S4xxxg",
      "node_name" : "node3",
      "transport_address" : "xxx.xxx.xx.xx:9300",
      "node_attributes" : {
        "ml.machine_memory" : "67190153216",
        "xpack.installed" : "true",
        "transform.node" : "true",
        "ml.max_open_jobs" : "20"
      },
      "node_decision" : "no",
      "store" : {
        "matching_size_in_bytes" : 64147690805
      },
      "deciders" : [
        {
          "decider" : "same_shard",
          "decision" : "NO",
          "explanation" : "a copy of this shard is already allocated to this node [[sessions2-200920][3], node[xxx3iYkaxxxpxxxLL_0Sxxx], [P], s[STARTED], a[id=xxxxTLXxxxxW2xcxjxtxHLUg]]"
        }
      ]
    },
    {
      "node_id" : "xxxGvfV3xxxmOZxxxw2bxxx",
      "node_name" : "node4",
      "transport_address" : "xxx.xxx.xx.xx:9300",
      "node_attributes" : {
        "ml.machine_memory" : "67190153216",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true",
        "transform.node" : "true"
      },
      "node_decision" : "no",
      "deciders" : [
        {
          "decider" : "shards_limit",
          "decision" : "NO",
          "explanation" : "too many shards [2] allocated to this node for index [sessions2-200920], index setting [index.routing.allocation.total_shards_per_node=2]"
        }
      ]
    }
  ]
}

From the output above the only way I can think of to resolve this is by moving a replica shard from its current node to node 3 and then moving unassigned to whichever node is available, expect for node 3.

4

Can you elaborate a little more on your allocation strategy? Why do you have a max of 2 shards per node? How many shards do you have in total?

5

@warkolm
1 primary and 1 replica per node so a total of 8 shards per index

6

Yes, I think this is the way to do it. You may however want to consider increasing the limit per node to at least 3 as I suspect you may end up with unallocated shards and reduced resiliency if a node failed with the current settings.

7

@Christian_Dahlqvist @warkolm I'm not sure how to exactly execute that. it appears that the reroute command would be used here but from the docs I don't see away to explicitly state that I want to move a replica and allocate_replica command appears to be for unassigned shards but I need to move a replica shard first then address the unassigned shard.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.