We currently have a 3 node VM cluster that was built as a POC. We are now wanting to move forward with a full physical implementation, but we have some cost limitations. Is it better to have a two node cluster built on beefy servers (384gb of ram and Intel Xeon processor) or a three+ node cluster with much less memory and a less powerful CPU? I am assuming with the two node cluster each index would have two shards and the three node cluster would have three shards.
So essentially, is it better to build the cluster tall with only a couple hosts or wide with lost of hosts?
Assuming that you have a fast and reliable network, lots of smaller hosts is recommended over fewer beefier hosts. One might argument that the latter would provide a better performance since elasticsearch nodes (you'll need more then one per host with a 384gb, it would ~6 nodes per host) will be communicating locally, but if you have a fast and reliable network, then it won't compensate the following disadvantages:
Availability is worse. It's 2 nodes, so if you loose one, half of your cluster is down.
Management problems. It's harder to manage many processes in a single host, specially elasticsearch nodes. You'll would need to use things like cgroups and other special configurations because without it troubleshooting problems will become a nightmare.
Bottlenecks. With a single host, hardware resources will be shared. Even though 384gb is lots of ram, what about CPU cores and I/O bus for example? You'll need lot's of elasticsearch nodes and elasticsearch is very resource hungry, so they will be fighting those shared resources and if you don't configure properly you will hit bottlenecks (also related to item 2).
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.