Data storage estimation for ES cluster

Elastic Stack Elasticsearch
#1

Hi All

We have requirement to build ES cluster for 10TB per day to retain the data for 60 days.

With the configuration of Replication shard to 2. Primary shard may be depending on the index receiving the daily volume which can vary.

  1. what is the data storage required for 10TB data raw data with replica set 2?

  2. Is it 10TB (primary shard) + 2 * TB (for replica shards) = 30TB the storage required in the ES cluster?

  3. Is there is any compression applied for shards? If yes, what will be space required for 10TB raw data (log files)?

  4. What is the best number of primary shard for the index which gets 1TB per day?

  5. If there is any formula to calculate the storage space required, please let me know.

  6. Planning to build ES node with 32GB RAM, 18 CPU cores. What should be shard size for the better performance?

Best Regards
JH

1 Like
Sizing Architecture for Huge amount data
#2

I would recommend this video on quantitative cluster sizing, as it describes how to go about answering your questions. There is no easy, analytical formula to plug numbers into, so you will need to benchmark in order to get an accurate answer.

1 Like
#3

thanks.. THe video is very basic and for beginners..

If anyone has done big scale deployment like my requirement, please feel free to suggest the options.

#4

The principles discussed in the video still apply to large use-cases. As a matter of fact, the larger the data volumes, the more important it is to go through a proper sizing exercise in order to identify the answers to your questions.

#5

I almost exclusively see 1 replica used for logging use cases, as the total data volume on disk is one of the factors that drive the cluster size. The ratio between raw data size and the size it takes up once indexed and replicated on disk depends on a number of factors, e.g. type of data you are indexing, how much data you during enrichment and what mappings you use. This blog post, which is getting a bit old, show an example of how mappings and index settings can affect this ratio.

Eklasticsearch applies compression, and as described in the blog post I linked to, there are a couple of options.

As stated in the video, we always recommend you perform a shard sizing benchmark to identify the ideal shard size. When I have benchmarked different types of data I have seen it range from a few GB to a few tens of GB.

No, there is not as it depends on a large number of factors as outlined above.

The amount of data and shards a node can handle typically depends on the heap size. We recommend a maximum heap size of around 30GB, which means nodes with 64GB of RAM seems to be the sweet spot. Another thing that is very important for the performance of Elasticsearch is the speed and throughput of the storage. We generally recommend local attached storage, but also have a lot of users that run on networked storage. The performance of networked storage can however vary a lot, especially since Elasticsearch performs a lot of reads and writes across the data files.

#6

Thanks for the details. Yes, the video gives some basic ideas for planning. I was expected that in the video they would have described one large platform deployments architecture & the hardware specs they used for that deployment.

I think i got have sufficient pointer now after reading few blogs, video, your comments & ELK documentation. I shall come up with better platform for this large data set requirement.

#7

There are a few different architectures that we use for large-scale deployments. It is possible to set up a cluster with all data nodes having the same specification, but for many use-cases with high ingest rates we see the hot/warm architecture being deployed with success. Depending on constraints around hardware and retention periods, one may be more suitable than the other. I work as a Solution Architect at Elastic, and we often help users with large use-cases work through this type of questions in order to get an estimate that is as accurate as possible and provide guidance on architecture and machine specifications.

#8

Thanks again for the details.

#9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.