I am setting up a new elasticsearch 6 cluster. Our cluster is going to be write heavy cluster.
We ingest roughly 1 TB of data each day
Config of the machines is as follows :
Disk : 1.5TB * 2
Memory : 256 GB
Cores : 40
I have read in blogs that Elasticsearch works best with heap less than 32 GB
I am thinking of two options here :
Divide each machine into 2 VMs
Each VM(node) will have 128 GB RAM, 30 GB heap for ES, 20 cores and 1.5TB
Let each machine be one node
Each node will have 256 GB RAM, 30GB heap, 40 cores and 3.0TB disk space
Kindly suggest what config would be the best for my cluster? I need to set this cluster on this weekend.
Kindly suggest asap