Recommended disk configuration for reasonable shard reallocation time

Elastic Stack Elasticsearch
1

I am estimating hardware configuration for customer environment but hesitating how should I configure the disk.

My reqirements are

2500 evt/sec (Maximum)
58 TB of logs ( holding for 1year )
5 shards, 1 replica for each index.

I am considering to split into 3 nodes which ends up with about 20 TB / node.
Considering the size of storage, I want to use 4TB 7200 RPM NL-SAS with RAID 1 + 0.

However, I assume that 20 TB of NL-SAS RAID 1 + 0 will require long recovery time
approximately ,

1491 hours (100 IOPS per disk , 5 disks per node )

May I ask for advice about how should I configure my node and disk in this case? I would appreciate any advice.

2

20TB of data on a single node is in my experience too much. It is however hard to determine exactly how much data a single node can handle. I would recommend watching the following video on cluster sizing to get an idea how to best determine how much data a node can handle through benchmarking.

3

@Christian_Dahlqvist

Thank you for the reply. Would you please describe the part too much ?

Do people generally keep the volume tight , perhaps <10TB, and increase the number of nodes when dealing with large volume of data?

4

I typically see less than 10 TB per node for logging use cases, but am not aware of any hard limit, as this depends a lot on the use case. The time it takes to move data around in case of node failure is part of this, but query patterns and performance also play a part. It is typically recommended to keep shard sizes below 50 GB, as larger shards can cause problems for recovery unless you have very good network bandwidth. Each shard also comes with some overhead, which means that a node can not handle an infinite number of shards.

5

Thank you for clear experience.

To be honest this deployment will be our first large volume deployment.
Would mixing SSD and Spiining disk using hot-warm architecture recommended solutin to deal with large volume data?

6

As you indexing load seems reasonable low, especially when spread out across the nodes in the cluster, I don't think a hot/warm archhitecture will buy you much. You need to optimize your mappings in order to optimize the size indices take up on disk and use the best_compression codec. Then you need to benchmark to see how many nodes you need.

7

Ok. Thank you.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.