One point I'd like to consider with you and ask you assistance is the hardware requirements, which we should use for the reliable and efficient operation.
This has likely been discussed many times. I'd like to get some advice before committing.
We have input data, give or take, as following:
EPS – 10000
Online Retention – 90 days
Offline Retention – 1000 days (3 years)
Total Raw GB/day – 600
The ELK stack will look like this:
3 nodes Elasticsearch cluster
1 server for Logstash with
Arcsight Smart Connector (Do I need a detached server?)
1 Kibana server
Would there be any chance of sharing the required hardware specification to deploy and implement ELK stack?
If we make the simplified assumption that your index data on disk take up the same amount of space as the raw data and that you will have a replica for high availability we get 1.2TB of indices generated per day. Over 90 days this is over 100 TB of data. Given that I would expect you to need considerably more than 3 Elasticsearch nodes.
If we were to assume that each node can handle e.g. 5TB if data you would need around 20 nodes. Whether this is a high or low estimate will depend on your hardware and performance requirements.
The amount of data a node can handle is usually driven by query latency requirements or heap usage. You can have a look at this webinar for a discussion around this. To optimize and reduce the size you indices take up on disk and ensure you do better than the simplified assumption used above I would recommend looking at this section in the docs.
It depends on size and complexity of events, retention period, query concurrency, query latency requirements, type of hardware and storage used. There is basically no easy way to give an accurate answer so I would recommend you test and run some benchmarks.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.