Elasticsearch moving a replica shard between nodes

Elastic Stack Elasticsearch
1

Hi,
I have a cluster with almost all nodes 111 shards and only 2 nodes 108.
I understand that ES balance the cluster with shard count.
cluster has around 19 nodes, 3 master eligible of them 2 data as well and 16 data nodes.
these two nodes with less shards are always reaching low watermark. Even though the shards are around same size.
I have few nodes with 600gb free and these two nodes with less shard count have only 70gb free. This is because of the few indices with very less size and few with more size.
ES version: 7.5.2
Questions:

  1. How to move replica shards from one node to another. (delete and direct to allocate them on a specific node or is there any direct way to issue a command using reroute).
  2. When rebalancing the cluster, how ES calculates what shard to move from one node to another.
2

That sounds fairly normal to me. See this section of the manual, in particular:

NOTE: It is normal for nodes to temporarily exceed the high watermark from time to time.

and

TIP: It is normal for the nodes in your cluster to be using very different amounts of disk space...

I don't think you should take any manual steps to address this.

3

Hi @DavidTurner
It is normal if its temporary.
But those two nodes are always in a low watermark level,
I am guessing this is because all the indexing always happening to these nodes because of the lesser shard count. (even if the rebalancing happening, because of the indexing to these nodes, they are reaching again the watermark levels).

Even If I create a no data index with multiple shards on that node only to increase the shard count, cluster is rebalancing (moving away other indices) and somehow still after some time these nodes still with full capacity and less shards in all.

  1. That is why I was trying to see how ES selects a shard to move in order to rebalance.
    MY ES version is 7.5
4

It's normal to temporarily exceed the high watermark, but it's also completely normal to exceed the low watermark on some nodes basically forever. This doesn't need any action.

The docs to which I linked earlier explain this, and the section just above explains how rebalancing works.

5

Ah I see that you're on an older version, which means it will be emitting a lot of misleading logs about watermarks. This was fixed in more recent versions:

Upgrade to ≥7.6.0 to avoid these noisy logs.

6

it's also completely normal to exceed the low watermark on some nodes basically forever -?

Yes, few nodes are always at low watermark, But doesn't it make those nodes slower in other operations?

Also,

  1. lets say I disable the rebalancing, what happens if that node reaches high watermark, will ES start indexing to the nodes with higher shard count?
  2. can you let me know on how to move a replica shard from one node to another.
7

I wouldn't have thought so, no. This is also covered in the reference I linked above:

A cluster with equal disk usage on every node typically performs no better than one that has unequal disk usage, as long as no disk is too full.

"Too full" is definitely above the high watermark (pretty much by definition: if you want to keep more space free for performance reasons then you should adjust the high watermark to suit)

This is also covered in the reference I linked above:

If a node exceeds the high watermark then Elasticsearch will solve this by moving some of its shards onto other nodes in the cluster.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.