We have a cluster that we want to add as much capacity to as possible.
We have some reservations on AWS that don't expire until a very long time from now, and we wan't to utilize those reservations in our cluster. However, we have also found that another instance type that has much more memory, is performing very well for our workloads and wan't to change all of our instance types to that type eventually. The CPU performance between the two instance types are identical.
Our setup consists of a lot of different indices for different purposes that do not necessarily have anything to do with each other, so many queries may end up only touching shards on either the low-mem nodes or the high-mem nodes, or both. Judging from that, my guess would be that it would not be an issue at all, and that some queries only touch the high-mem instances would just be performing really well, while those that happen to touch a shard on the low-mem would just be constrained by the worst performing node.
Does that make sense?
Are there any stability issues that should be taken into concern when running a couple of nodes with less memory than the rest of the nodes?