Sizing ELK cluster for 12GB daily logs

Elastic Stack Elasticsearch
1

Hello,

I am moving an ELK demo deployment (FileBeats -> Logstash cluster -> ElasticSearch cluster) to a production one. I am expecting ~12GB of daily logs to be fed into ElasticSearch, with a retention period of 1 year. I would like to ask about the sizing of the overall solution, and here is what I am thinking of:

  1. 2 x Logstash nodes (8 GB, 8 vCores each)
  2. 3 x ES Master nodes (6 GB, 4 vCores each)
  3. 3 x ES Data nodes (32 GB, 12 vCores each)

Questions:

  1. Are the above adequate as a starting point? Can I safely reduce them?
  2. Do the specs of each node make sense for the expected amount of traffic? Are they over-sized?
  3. Do I need 3 ES Master nodes?
  4. Is a load-balancer required in front of ES nodes?

Thank you,
Gerasimos

2

If we make the assumption that you logs will take up the same amount of space on disk as the raw form and that you will have one replica, that gives around 8.5TB of data on disk in total. You can likely reduce this by optimizing your data. That is almost 3TB per data node, which should be fine given your specification assuming you follow best practices.

I think this looks like a good starting point. Since you are indexing relatively little data per day you may be able to make your data nodes master-eligible and might not require the dedicated master nodes.

Memory-wise I think this looks good, but I suspect you could reduce the amount of allocated CPU if you wanted to. You can probably half it across the board.

Not necessarily. Dedicated master nodes are typically added for larger clusters or if the cluster is under heavy load and therefore suffering from instability.

No, that is generally not required.

If you are going to use Kibana, you may want a host for that as well. It is often deployed together with a coordinating-only node which acts as a load-balancer.

3

Thank you Christian!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.