Sizing Elasticsearch

Hi everyone,

I want to plan the ES cluster reasonably:

  1. I have 15 nodes and each nodes have 125G RAM and 31 CPU;
  2. The total size of data is 30T(include replica);
  3. The total count of shards is 7100(include replica);
  4. Es version is 6.8.0;
  5. index and search are all frequently;

How do I size my cluster? (how many master node do i need? Whether several coordinate nodes are required)

Best Regards!

It sounds like you have already sized your cluster and have it up and running. How is it performing?

You generally should have at least 3 master-eligible nodes, but 3 is often the optimal number. If your cluster is deployed across a larger number of availability zones/racks you may increase this.

This depends a lot on the use case, as coordinating-only nodes are often not required. If you have a lot of querying you can add this type of node to see if it improves performance. Set the number so they are under reasonable load.

@Christian_Dahlqvist Yes, the cluster had been running for some time before I took over,and because the cluster's configuration is default value, query efficiency is not ideal.

I want to optimize the cluster by resizeing the cluster.

But I found that although I knew some theory, I still didn't know how to start in practice…

for example:
Does 125G ram is too big to a node(es node's jvm can not cross 32G)? is it better to deploy multiple es instances on one node?

Your setup looks reasonable. Have you identified what is limiting performance? Is it CPU? Disk I/O and utilisation? Heap pressure and GC?

What does the workload look like?

Hi, @Christian_Dahlqvist

the workload now looks good:

but sometimes the query efficiency is not good.

Do I need to set up extra ES instances on each node? I think 125GB is too much big.