Is it possible to distribute unequal loads among the unequal servers (different processing power)

Elastic Stack Elasticsearch
1

Is it possible to distribute unequal loads among the unequal servers (different processing power)?
Assume we have an Elasticsearch cluster with different nodes performances. When we create an index in this cluster, all the nodes will have the same number of shards, while some nodes have stronger CPUs and more memory and can handle more shards.
This problem is highlighted when we use Esrally to get the performance of the cluster. For an example assume this command:

esrally --offline --track=eventdata --target-hosts=m1:port1,m2:port2,m3:port3,m4:port4 --pipeline=benchmark-only --challenge=append-no-conflicts --track-params='number_of_shards:4, number_of_replicas:0'

In the above example, 1 shard is assigned to each machine. If we assume m1 and m2 are two times more powerful than m3 and m4, then we should assign 2 shards for m1, 2 shards for m2,1 shard for m3, and 1 shard form4; totally 6 shards, maybe a command like this:

esrally --offline --track=eventdata --target-hosts=m1:port1:'number_of_shards:2',m2:port2:'number_of_shards:2',m3:port3:'number_of_shards:1',m4:port4:'number_of_shards:1' --pipeline=benchmark-only --challenge=append-no-conflicts --track-params='number_of_shards:6, number_of_replicas:0'

2

Please reply to my question. :disappointed:

3

Elasticsearch assumes all data nodes have the same specification, so having mixed clusters can cause performance problems and hotspots. You can divide the cluster into zones through shard allocation filtering and have different hardware per zone but data will not be automatically distributed then.

4

Thank you for your clear answer. Is there any good link that I can read more about elastic zones including a clear example?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.