One of the beautiful things about elastic is it's automatic shard
allocations. This is also something that is painful, when reallocating
large amounts of data - at least from my (lack of) experience.
I have a cluster with 3 nodes. All indexes have replication set to one.
Let's say I turn off a node, migrate it to a system with clean disks,
bring it back online. What happens next is a bit of voodoo.
Presumably the two active nodes start replicating their shards to the clean node. That does indeed happen, however they also start cross replicating. So instead of both nodes reallocating to the node with the fewest shards, it seemingly tries to cross balance amongst the two with the most shards as well.
To visualize, elastic01 and elastic03 were the active nodes. Elastic02
came up clean.
elastic01 -> elastic02
elastic01 -> elastic03
elastic03 -> elastic02
elastic03 -> elastic01
elastic03 -> elastic01
elastic01 & elastic03 have about 400 shards each, but instead of them
just sending some to elastic02, what I see is elastic trying to also
balance the shards between elastic01 and elastic03, especially when the
shard count between them starts to differ do to different sized shards.
I tried to set balance.threshold to 30 so they wouldn't cross balance,
but that had no effect that I could discern.
Hopefully this isn't too convuluted to follow. How can we get elastic
to prioritize the shard reallocation to the node with the fewest shards
rather then clogging up the process with cross node replication?
Vince