Filebeat / kubernetes / data loss

I am using following config https://raw.githubusercontent.com/elastic/beats/master/deploy/kubernetes/filebeat-kubernetes.yaml plus settings symlinks: true

I have container'd containers with rotated logs by default. 5 log files 3 of them are *.gz Files rotated by size exceeding 10 Mb. I have 30 harvesters and now i have 10000 d/s. Many containers write multiple logs per second

I observe data loss. Read offset is larger then file size.

I think I should use these parameters but I am confused which values should I set them to.

close_inactive: 1m
clean_inactive: 10m
ignore_older: 5m

Hey @elastic_user1,

Could you explain more about the rotation of your logs? Are they being rotated with copytruncate?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.