I need guidance on setting up an ES cluster which can consume ~1TB of logs per day and storing them for 30days.
Here is what I am thinking to do by procuring Basic license:
- Elastisearch - 3 master nodes (x-pack and tls enabled) running on kubernetes (aws EKS), I have no idea about how much heap so will start with 10G heap and will increase gradually as needed. 1 replica of each index.
- ES storage - gp2 EBS for starting, its max size is 16TB, I may need to increase the master nodes to accommodate more data if disks are getting full.
- fluentd daemonset configured to send logs from k8 to ES (this is straight forward)
- Kibana running on k8 with 2G heap to start with.
Can anyone guide me if I need to do the things differently here?