I am using following config https://raw.githubusercontent.com/elastic/beats/master/deploy/kubernetes/filebeat-kubernetes.yaml plus settings symlinks: true
I have container'd containers with rotated logs by default. 5 log files 3 of them are *.gz Files rotated by size exceeding 10 Mb. I have 30 harvesters and now i have 10000 d/s. Many containers write multiple logs per second
I observe data loss. Read offset is larger then file size.
I think I should use these parameters but I am confused which values should I set them to.
close_inactive: 1m
clean_inactive: 10m
ignore_older: 5m