Guidance on setting up elasticsearch to handle terrabytes of logs

Elastic Stack Elasticsearch
1

I need guidance on setting up an ES cluster which can consume ~1TB of logs per day and storing them for 30days.

Here is what I am thinking to do by procuring Basic license:

  1. Elastisearch - 3 master nodes (x-pack and tls enabled) running on kubernetes (aws EKS), I have no idea about how much heap so will start with 10G heap and will increase gradually as needed. 1 replica of each index.
  2. ES storage - gp2 EBS for starting, its max size is 16TB, I may need to increase the master nodes to accommodate more data if disks are getting full.
  3. fluentd daemonset configured to send logs from k8 to ES (this is straight forward)
  4. Kibana running on k8 with 2G heap to start with.

Can anyone guide me if I need to do the things differently here?

2

Might be worth taking a look at this blog

Recently we have changed Warm ratio to 160:1

Also might take a look at this

1 Like
3

I would also recommend you look at these webinars:

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.