Filebeats 7.4.0 running in a relatively large and busy Tanzu Kubernetes cluster (1.20.x) The Filebeat registry file /var/lib/filebeat-data/registry/filebeat/data.json grows causing disk throttling as the file size gets in the 25-50MB range.
To my understanding the file is being written to constantly by the dozens of pods on the node, so as the file size grows, that's putting strain on our storage controllers.
Our fix, has been to stop the filebeat PID on the node, delete data.json and kill the PID to restart the process. That's a band-aid for a few weeks, but the file size slowly creeps back up.
Example, here are a few file sizes and their increase over the past 2 days.
node 3/29 3/30
1 638k 896K
2 541k 707K
3 850K 1.1M
4 518K 645K
5 510K 709K
6 672K 863K
7 650K 889K
data.json growing about 200K per day.
I've tried implementing the clean_inactive & clean_removed settings, as well as clean_*, with no visible impact.
Any additional input would be greatly appreciated!
Thank you!
