Hardware Sizing for ELK stack

Elastic Stack Elasticsearch
1

Hi All

We decided to use ELK for our log analysis and i have been using it in my laptop for 3-4 weeks now and we do mostly visualizations for Apache and IIS web server logs.

We intend to take this to production and i need to come up with the hardware configuration.
The log data inputs are as follows.

  1. around 10-12 GB of log data is produced everyday
  2. We need to retain dashboard data for not more than 15 days.
  3. We don't do any specific search on elasticsearch just input log entries and we do charts out of them
    we do avg response times , total hits and various pie charts , that's about it.

can you please give me some points in terms of memory , disk storage and cpu requirements.

UPDATE -- got an update saying we will be monitoring logs for around 50 apps each generating around 10 GB , so we are talking about processing 50 GB of data and the other requirements are the same.

Thanks

2

Hi,

Hard to tell without testing but, if you are on AWS, you may be OK with a pair of i2.something instances for data nodes and 3 m3.medium dedicates masters. But the best way is to test. Use something like SPM ( http://sematext.com/spm/elasticsearch-monitoring.html ) on your nodes, start indexing and observe metrics/capacity. Then run queries with something like JMeter and again observe metrics/capacity. And tune before you buy more hardware/EC2 instances. :slight_smile:

Otis

Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/

3

Thanks for the reply.

I guess i will try and get more realistic requirements in terms of how many bytes of log data i need to parse , frequency and also requirements w.r.t time duration for storage of log entries in ES.