Hardware requirement ELK

Dear Elk support team,

One point I'd like to consider with you and ask you assistance is the hardware requirements, which we should use for the reliable and efficient operation.
This has likely been discussed many times. I'd like to get some advice before committing.

We have input data, give or take, as following:

EPS – 10000

Online Retention – 90 days

Offline Retention – 1000 days (3 years)

Total Raw GB/day – 600

The ELK stack will look like this:

  • 3 nodes Elasticsearch cluster

  • 1 server for Logstash with
    Arcsight Smart Connector (Do I need a detached server?)

  • 1 Kibana server

Would there be any chance of sharing the required hardware specification to deploy and implement ELK stack?

If we make the simplified assumption that your index data on disk take up the same amount of space as the raw data and that you will have a replica for high availability we get 1.2TB of indices generated per day. Over 90 days this is over 100 TB of data. Given that I would expect you to need considerably more than 3 Elasticsearch nodes.

If we were to assume that each node can handle e.g. 5TB if data you would need around 20 nodes. Whether this is a high or low estimate will depend on your hardware and performance requirements.

The amount of data a node can handle is usually driven by query latency requirements or heap usage. You can have a look at this webinar for a discussion around this. To optimize and reduce the size you indices take up on disk and ensure you do better than the simplified assumption used above I would recommend looking at this section in the docs.

What hardware are needed for Logstash for eps 10000 and Kibana ?

It depends on size and complexity of events, retention period, query concurrency, query latency requirements, type of hardware and storage used. There is basically no easy way to give an accurate answer so I would recommend you test and run some benchmarks.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.