Hardware requirement for elasticsearch

[Nusrath (Syed Nusrath)]

I'm trying to setup elasticsearch cluster. I will get maximum of 20TB of data.

So what will be hardware required to set up ElasticSearch 6.x and kibana 6.x

• Which is better Elastic search category –Open source/ Gold/Platinum
• What is ideal configuration for server- side RAM/Hard disks etc.
• Tool used to monitor ES performance

Appreciate your help!!

[Christian_Dahlqvist]

You have not given any details about your data or use-case or expected indexing/query load, so I suspect it will be very difficult for anyone to give a good answer.

[Nusrath (Syed Nusrath)]

Thanks for reply Christian_Dahlqvist,

Sorry but i don't know what is indexing/query load is.

Basically what we are trying is simply getting data and inserting into elastic search and showing dashboards in kibana.

We have already developed POC on small chunk of data but now we are expecting nearly 20TB of data.

So to work on 20TB of data , what are the machine or hardware requirements ?

[Christian_Dahlqvist]

In order to get some kind of answer you need to describe your use-case and provide an indication of how much data you intend to index every day. If you have a logging and/or metrics use case with time-based data, this will typically depend on the expected retention period.

I would recommend you have a look at the following resources:

How many shards should I have in my Elasticsearch cluster? | Elastic

Sizing Hot-Warm Architectures for Logging and Metrics in the Elasticsearch...

https://www.elastic.co/webinars/optimizing-storage-efficiency-in-elasticsearch

https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right

https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing

It would also be useful to know where you intend to deploy this and whether there are any constraints around hardware that should be considered when making recommendations.

[Nusrath (Syed Nusrath)]

Thanks Christian,

I have gone though above links and elastic use cases and we observe our use case will be Log monitoring and analysis using dashboards with full text search.

In our case it would be Log analysis, where we will input data on a daily basis (~appx 1 TB per day ) and run searches on them this process will be continuing for next 2 years and I think we should go with Hot-warm architecture.

Can you suggest me what are the hardware requirements to set up ES cluster along with kibana?

Thanks

[Christian_Dahlqvist]

If you want to ingest 1TB per day and keep it searchable for 2 years, that will be a large cluster (or even multiple clusters). Given the scale I would recommend you run some tests to determine how much space your type of data will take up on disk and how much data your nodes will be able to hold while still meeting search latency requirements. This will allow you to more accurately estimate the size of the cluster(s) required than anyone here can provide. The links I provided earlier should guide you on how to do this.