ES Cluster Capacity Planning - Storage Limit

Elastic Stack Elasticsearch
1

In elasticsearch documentation for capacity planning, I read Per GB Heap will handle the 20 shards, with avg shard size of 20 - 40 GB.

In ES Cluster it has 10 data nodes with the following config,

64 GB Heap (30 GB heap-allocated)
16 CPU's
2 TB SSD

What I would like to know based on the capacity planning docs,

It's illustrated data storage per Data Node ( 30 GB heap allocated to ES )

30 GB * 20 Shards ( Each 20 - 40 GB) . = 600 Shards , Approx Disk size ( 12 TB - 24 TB ) Per node.

How much TB we can have stored max per ES Data node? , Do we have any limit on this?

I handled up to 5 TB per data node using the HDD .

Does anyone store high storage size per data node? (Real-time or benchmark)

2

The 20 shards per GB heap is a recommended maximum. The total amount of data a node can handle is generally limited by heap or query performance. There is no guarantee that a node can handle that many large shards. Have a look st this webinar for a discussion on the topic.

3

Thank You, @Christian_Dahlqvist

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.