In our cluster we have a problem where the nature of the data means that some indices will have 80 primary shards each 10gb large, which other indices will have a single primary shard that is only 500mb.
My experience with Elasticsearch has been that this can lead to the cluster making some poor balancing decisions, as it seems to want to treat all shards as equal. Recently a new node joined our cluster and it ended up being given a disproportionate number of the large shards. The end result was that this node had just over half the number of shards as every other node in the cluster (which all only vary by a single shard) but over twice the storage usage (enough to push it past our low watermark).
There are some things we do to combat this (we are already set up with forced awareness, and we do not have problems with data loss). We've tweaked the index balance weights to favor an even distribution of indices over shards. And all of our indices are set up to allow no more shards per node than is necessary for our cluster.
But neither of these solutions work well when you have new indices created on an hourly or daily basis. Because even if every index is only allowed to have a single shard on each node, if you have an hourly index over several days a single node could end up with dozens of shards from essentially the same index.
I'm looking for a better solution. Re-indexing those hourly indices into daily or daily into weekly would help, but it is not really an option for us at the moment. Re-indexing is very expensive and our current infrastructure does not really allow it. Our smallest shards are already as big as we can make them, and our larger shards are not even up to the size most people online seem to aim for. Despite that, we could increase those shard counts even further to get the size variance down.
What I would really like is a way to configure balancing rules across multiple indices (or an alias). To be able to say that: "Between index x, y, and z there can only be one shard per node."