Filebeat Registry File Growing

Elastic Stack Beats
1

Filebeats 7.4.0 running in a relatively large and busy Tanzu Kubernetes cluster (1.20.x) The Filebeat registry file /var/lib/filebeat-data/registry/filebeat/data.json grows causing disk throttling as the file size gets in the 25-50MB range.

To my understanding the file is being written to constantly by the dozens of pods on the node, so as the file size grows, that's putting strain on our storage controllers.

Our fix, has been to stop the filebeat PID on the node, delete data.json and kill the PID to restart the process. That's a band-aid for a few weeks, but the file size slowly creeps back up.

Example, here are a few file sizes and their increase over the past 2 days.

node 3/29 3/30
1 638k 896K
2 541k 707K
3 850K 1.1M
4 518K 645K
5 510K 709K
6 672K 863K
7 650K 889K

data.json growing about 200K per day.

I've tried implementing the clean_inactive & clean_removed settings, as well as clean_*, with no visible impact.

Any additional input would be greatly appreciated!

Thank you!

2

Welcome to our community! :smiley:

Please note that version is EOL and no longer supported, you should be looking to upgrade as a matter of urgency. There are always fixes for issues in newer version and while I can't guarantee that's a solution for this particular problem, you are running a release from nearly 4 years ago!

Hopefully someone else can add some further comments :slight_smile:

3

You might be running into this issue. The recommended solution, would require you to upgrade as it is not available on the version you're using.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.