Nodes with different amount of memory

[Silas_Hansen]

Hi

We have a cluster that we want to add as much capacity to as possible.
We have some reservations on AWS that don't expire until a very long time from now, and we wan't to utilize those reservations in our cluster. However, we have also found that another instance type that has much more memory, is performing very well for our workloads and wan't to change all of our instance types to that type eventually. The CPU performance between the two instance types are identical.

Our setup consists of a lot of different indices for different purposes that do not necessarily have anything to do with each other, so many queries may end up only touching shards on either the low-mem nodes or the high-mem nodes, or both. Judging from that, my guess would be that it would not be an issue at all, and that some queries only touch the high-mem instances would just be performing really well, while those that happen to touch a shard on the low-mem would just be constrained by the worst performing node.

Does that make sense?
Are there any stability issues that should be taken into concern when running a couple of nodes with less memory than the rest of the nodes?

[Christian_Dahlqvist]

Elasticsearch by default assumes all nodes have the same specification and will try to distribute data evenly. This could mean that lower memory node would hold the same amount of data as the more powerful ones and therefore potentially suffer from heap pressure.

One way to get around this could be to use shard allocation filtering to create two "zones" in the cluster, one with all lower memory nodes and one with all high memory nodes. These zones would basically be a cluster within the cluster and you can allocate indices to one or the other and that way control how much the different data node types hold.

[Silas_Hansen]

That's a really good tip - thanks!