Is it possible to distribute unequal loads among the unequal servers (different processing power)?
Assume we have an Elasticsearch cluster with different nodes performances. When we create an index in this cluster, all the nodes will have the same number of shards, while some nodes have stronger CPUs and more memory and can handle more shards.
This problem is highlighted when we use Esrally to get the performance of the cluster. For an example assume this command:
esrally --offline --track=eventdata --target-hosts=m1:port1,m2:port2,m3:port3,m4:port4 --pipeline=benchmark-only --challenge=append-no-conflicts --track-params='number_of_shards:4, number_of_replicas:0'
In the above example, 1 shard is assigned to each machine. If we assume m1 and m2 are two times more powerful than m3 and m4, then we should assign 2 shards for m1, 2 shards for m2,1 shard for m3, and 1 shard form4; totally 6 shards, maybe a command like this:
esrally --offline --track=eventdata --target-hosts=m1:port1:'number_of_shards:2',m2:port2:'number_of_shards:2',m3:port3:'number_of_shards:1',m4:port4:'number_of_shards:1' --pipeline=benchmark-only --challenge=append-no-conflicts --track-params='number_of_shards:6, number_of_replicas:0'