Hi!
I'm using ELK (Elasticsearch, kibana, filebeat, metricbeat) deployed via Helm.
I'm collecting small amount logs from a few containers and PVC for Elasticsearch is set to 2Gi.
For some reason this space after one night gets filled and kibana isn't able to show any logs and k logs elasticsearch-master-0 shows:
Caused by: java.io.IOException: No space left on device
in the stack trace. And those PV are filled +95%. After one day to make my stack work I have to clean PVC and restart deployment.
How can I solve/debug this issue? What is actually being stored there so quickly? Is there an option for yaml configuration to make elasticstack drop the oldest content of it's storage to avoid this issue and make ELK stack run for longer time?
Thank you for any guidance and tips.