How are shards chosen when disk usage exceeds high water mark?

YuWatanabe · December 31, 2018, 3:07am

I appreciate if I could get help to understand behavior of disk high watermark in elasticsearch.

From the document, I have understood that when disk usage of the partion where path.data exists exceeds high watemark then shards will begin to be relocated to other nodes.

Controls the high watermark. It defaults to 90% , meaning that Elasticsearch will attempt to relocate shards away from a node whose disk usage is above 90%

But how will the shards chosen that needs to be relocated ? Will be chosen from the largest ones ?

I came across with this code , but couldn't find the part where it decides the shard.

DavidTurner · December 31, 2018, 8:43am

The order is undefined, and no effort is made to move the largest or smallest shards first. Here is the loop that goes through the shards checking whether they can remain:

github.com

elastic/elasticsearch/blob/022726011ce1d9e4748fd39293a3aeae6954b3ac/server/src/main/java/org/elasticsearch/cluster/routing/allocation/allocator/BalancedShardsAllocator.java#L626-L628


for (Iterator<ShardRouting> it = allocation.routingNodes().nodeInterleavedShardIterator(); it.hasNext(); ) {
    ShardRouting shardRouting = it.next();
    final MoveDecision moveDecision = decideMove(shardRouting);

And here is the decider:

github.com

elastic/elasticsearch/blob/022726011ce1d9e4748fd39293a3aeae6954b3ac/server/src/main/java/org/elasticsearch/cluster/routing/allocation/decider/DiskThresholdDecider.java#L256-L309


@Override
public Decision canRemain(ShardRouting shardRouting, RoutingNode node, RoutingAllocation allocation) {
    if (shardRouting.currentNodeId().equals(node.nodeId()) == false) {
        throw new IllegalArgumentException("Shard [" + shardRouting + "] is not allocated on node: [" + node.nodeId() + "]");
    }
    final ClusterInfo clusterInfo = allocation.clusterInfo();
    final ImmutableOpenMap<String, DiskUsage> usages = clusterInfo.getNodeLeastAvailableDiskUsages();
    final Decision decision = earlyTerminate(allocation, usages);
    if (decision != null) {
        return decision;
    }


    // subtractLeavingShards is passed as true here, since this is only for shards remaining, we will *eventually* have enough disk
    // since shards are moving away. No new shards will be incoming since in canAllocate we pass false for this check.
    final DiskUsage usage = getDiskUsage(node, allocation, usages, true);
    final String dataPath = clusterInfo.getDataPath(shardRouting);
    // If this node is already above the high threshold, the shard cannot remain (get it off!)
    final double freeDiskPercentage = usage.getFreeDiskAsPercentage();
    final long freeBytes = usage.getFreeBytes();
    if (logger.isTraceEnabled()) {

This file has been truncated. show original

YuWatanabe · January 7, 2019, 2:05am

I see. I will mark your comment as solution.

system · February 4, 2019, 2:05am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.