Number of nodes for primary and replicas required for resilience

Elastic Stack Elasticsearch
1

I am building a large elasticsearch cluster. I will have probably have around 100 to 150 primary shards. Since the dataset is very large, I am planning to limit the number of shards per node to max of 3.

So for 100 primary shards, with 2 replicas, there will be a total of 300 shards. So 100 nodes will be able to hold that. In this setting, how many nodes I can afford to loose ?

2

With 2 replicas you can afford to lose at most 2 nodes. If you lost more than two nodes then there's a chance that three of them contained the primary and two replicas of a shard.

However this seems like a very strange setup. Can you explain how you have determined that 3 shards per node is optimal? Normally you want a lot more than that.

3

We estimate that each primary shard will be around 75GB to 100 GB. Total size of the index will be 10 TB ( excluding replicas). I read an article and it says that for growing index with large dataset, limit the shard per node to 3. The application is heavy on indexing ( millions of documents per day) and also need to support 100,000 text search on the text documents.

Also, Does number of shards per node also depends on the RAM available on the node ?

4

Maybe I'm misunderstanding, but this doesn't sound like a 100-node-cluster job to me. There are clusters out there that index multiple TBs per day that contain nowhere near 100 nodes. Is 10TB the daily volume or the total? Did you mean PB and not TB?

The article you linked doesn't really spell out the details of their assumptions, but I suspect they don't apply in your case and/or they're trying to sell you a larger cluster than you need and/or they are 5 years out of date. I suggest you invest some time and effort into proper benchmarking of something closer to your actual workload, since that's the only real way to size a cluster properly. You might find it saves you a lot of money.

1 Like
5

The total volume excluding replicas will be around 10TB . it is not a daily volume. Daily volume will be around 3-4 GB. But there will be daily creates and updates of around 100 million very small text documents.

You're correct. Lot of articles on the internet have lot of outdated information. So based on that I arrived at the conclusion that 75GB to 100 GB is my ideal shard size. I guess, we can determine this only based on the results of benchmark.

I have already installed Rally using the docker image. Done few tests. I don't have the box ready to do the benchmarking. I am in the initial phase of planning.

Thank you for the replies. Really helps.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.