We are running a 50 node cluster in production. Each node has the following configuration -
Ram : 256 GB
Disk : 5.6 * 2 = 11.2 TB
Cores : 80 vCPUs
Heap : 29 GB
GC algo : G1Gc
We index roughly 2 billion documents a day and all our indices are daily indices and we keep 30 days of data in the cluster.
Each node holds around 4 TB of data with around 500 shards
Write/Read ratio : 90/10
Most of our search queries are aggregation queries and indexing requests are bulk indexing requests
With increasing load on the cluster, we are seeing lot of GC happening in the cluster. According to this blog, https://www.elastic.co/blog/a-heap-of-trouble, what heap size should we use for our cluster
I would recommend sticking with the 29GB heap but run multiple nodes per host if you are having problems with heap pressure. If you want a bit more separation you could consider using containers.
I tried running 2 VMs per machine, but after doing that, I am seeing performance degradation in indexing. Indexing speed has reduced to 1/5th to 1/2 on different VMs compared to BM. So, splitting machine into multiple nodes isn't helping out.
BM seems to outperform by a huge margin compared to VM
Because of the performance impact, we want to stick to only one node per host, what heap settings would you recommend?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.