Cluster Achitecture

[ysl]

Hello,

I would like to setup my Elasticsearch cluster with the hardware specification and condition as below:

Hardware for each node
16 Gb ram
8 virtual cores

conditions
Starts with 10Tb disk space in total
Data storage increase 1Tb per day
One single index contains daily data around 100Gb

Can anyone suggest the architecture that is optimum or good to start with, for example:

1. index slicing (is the size of 100Gb per index good to support daily data?)
2. number of shards ?
3. number of cores ?
4. are 10gb a suitable size to start with ?

Thanks in advance

[Christian_Dahlqvist]

How many nodes are you looking to have? How long do you intend to keep data in the cluster? What type of storage do you have?

To know how much a node can handle for your specific hardware, I would recommend running some benchmarks as outlined in these resources:

https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing

https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right