We have been evaluating ES as a migration path to simplify our proprietary search layer. Up until this point, we have been using a set of virtual machine nodes for our ES tests. We are about to deploy a few dozen blades as our initial set of dedicated ES nodes. If all goes well, we will be adding a second set of nodes to the cluster in 6-8 months.
A few of our tables are fairly large. The larger and simpler ones like logs we'll be keeping in Cassandra, but there are still a few tables in the 50-100 million docs range like products we need to index in ES. I have a couple questions for those who may have experience with ES migration or knowledge of the internal workings of ES regarding the performance of storage load balancing in ES. We want to have a better idea on our replication settings and whether (or how much) we need to spec-match the second set of machines to the initial set of machines in terms of storage capacity.
How well/quickly do the shards spread out across nodes when a large number of new nodes are added to the cluster? If the cluster contains two types of machines with different storage capacities, does ES load balance the storage accounting for the differences? If so, how well does ES do in this regard?