Three bigger data nodes or six smaller?

Elastic Stack Elasticsearch
1

Hi,

I have the following dilemma. I've been provided with 3 hosts on which to run a virtualized Elasticsearch cluster. The problems is that if I want to run 3 masters and 6 datanodes each of the elasticsearch nodes ends up with just 2 vcpus and 16GB of RAM. Alternatively, I can run 3 masters and 3 datanodes thus ending up with more vcpus and more RAM per node, but less nodes in total.

  1. Which is better in this case? More but less powerful nodes, or less but more powerful nodes?
  2. Is 2 vcpus enough for elasticsearch node?
  3. Will 8GB RAM for masters and 16GB for datanodes be enough?

I know I have to test the setup in order to know, but still any rough estimates will be of help.
I will ingest between 300-500GB a day.

2

Depends :stuck_out_tongue:

How large ate the physical nodes?

3

I have 3 hosts with 48GB RAM and 6 cores each.

The question is whether to put 1 master and 2 datanodes on each host. Or 1 master and 1 datanode. As said 300-500GB a day with no or little ingest transformation. I plan for the datanodes to do the ingesting.

4

I'd say 1 master, 1 data.

1 Like
5

Okay, then I will consider running with 3 masters (2 cores 16 GB) and 3 datanodes (4 cores, 32 GB). Does it makes sense then to change the defaults 5 shards/1 replica? How many shard will be best then? 3 shards and 1 replica?

Also I assume it is alright for the masters and data to have different heap sizes.

One last question, should I make master and data nodes equal in terms of RAM and CPU if I can or should I give the datanodes more resources?

6

Your masters do not need to be that big, I wouldn't bother with anything more than 3-4GB of heap for them (ie 6-8GB RAM).

Depends, but 3 makes sense.

Give the data node more.

7

Okay, thanks! One last question (again):

Do you think 2 vcpus per master node will be enough?

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.