How does Elasticsearch Splitting an Index Work?

[Christian_Dahlqvist]

@Hao_Yellow

Based on the understanding above, first of all the 100GB shards on node 3 will be copied 4 times, which add up to 400GB... So the first step, cloning, is a problem. Neither of the nodes can accommodate another 100GB, not to say 400GB.

It is indeed a problem and it is futher complicated as the delete by query will require a lot of additional disk space, as described in David's response:

@DavidTurner

Splitting the index works by cloning all the shards (multiple times) and then effectively running a delete-by-query on them, which certainly increases the reported size until merging cleans up the deleted docs.

@Hao_Yellow

And what about shard allocation? This process may be implicit in the cloning. If the operation is clone and delete-by-query, what will happen if neither nodes are not sufficient to contain the original shard of 100GB before delete-by-query?

I do not think the new shards can get allocated to another node until the delete by query has completed on the original node. Either way it does not work as transferring a 100GB before delete by query runs is not possible.

@Hao_Yellow

But I guess it's supposed to work anyway? And Elasticsearch is intelligent enough to work that out?

No. Splitting an index without the required additional disk space will not work.