We are trying to introduce tiering where we have a number of machines with small but fast disks and others with large but fast disks and we see that the nodes with small filesystems get swamped. At least in 1.5. We intend to update to 1.7 soon but I don't suppose that that will help.
Looking at BalancedShardsAllocator.java the code that seems to determine distribution is:
public float weight(Operation operation, Balancer balancer, ModelNode node, String index) {
final float weightShard = (node.numShards() - balancer.avgShardsPerNode());
final float weightIndex = (node.numShards(index) - balancer.avgShardsPerNode(index));
assert theta != null;
return theta[0] * weightShard + theta[1] * weightIndex;
}
Basically the intention is to get an even number of shards on every node and an even number of shards in each index on each node depending on the values in theta (properties cluster.routing.allocation.balance.{shard,index}).
As we have asymmetric node the shard balancing seems wrong and I was planning to reduce this theta to close to zero and boost the index theta or reduce the threshold.
Any comments?
/Michael