Logging Cluster Architecture

Elastic Stack Elasticsearch
1

Dear all,

I want to setup a logging cluster with ES. The ingestion should be 2000 Event per Seconds, meaning 160GB per day (hypothesis: 1 log is 1kb). The retention time is 365 days and the hot retention time is 15 days.

I took the following hypothesis but I'm not sure I'm right:

  • 1 index for all logs;
  • nearly 650 shards: the whole data (without replica) (160GB*365) with an overhead of 1.10 for indexing and 100GB per shard.

The whole data amount will be 133TB (with 1 replica) and I count a total of 26 warm nodes and 1 hot nodes (with 5TB storage per node). I choose 5TB storage per nodes according to the 40:1 and 100:1 ratio between RAM and disk for hot and warm nodes.

The RAM for hot nodes will be 128GB and for warm nodes 64GB. There will be nearly 24 shards per nodes.

To summarize, my architecture for 2000 EPS, 1 year retention and 15 days hot logs is composed of:

  • 1 hot nodes with 128G RAM, 5TB disk;
  • 26 warm nodes.

(I do not count there the master-nodes).

Is-it correct or oversized ?

Is-it possible to compress the data ? If I compress the data on warm nodes, I just have to take 20% of the data, so the architecture will only be composed of 5 warm nodes. Correct ?

Thanks a lot,

Laurent

2

Hi there,

just an up !

Laurent

3

Some comments:

You need to keep disk under 85%, so if you're going to store 5Tb per node, you need something like 6.25Tb disk (formatted)

You can't replicate your hot data with just 1 hot node. I'd recommend 3 or 4; with 1 node, no replica, with 2, you can't recover shards after a node failure. You can increase the hot retention to make efficient use of the disk space.

You may be able to store more than 5Tb on each node, but that becomes somewhat of a try and see. If you have "capacity on demand", so you can add hardware without delay when needed, I'd be tempted to try 4 hot nodes and 16 warm with disk that could hold 7-8Tb of data.

Use ILM to roll you indexes over by size, not daily, that will reduce the shard count. Roll at 40-50G or max weekly, for example.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.