Filebeat memory issues

Elastic Stack Beats
1

image
image.png1835×268 176 KB

Here's a (not very) pretty picture - the filebeats in our K8s cluster appear to start up, then over not-very-long increase their memory consumption until they crash with out of memory, then restart.

This is version 6.6.2, with a memory limit of 500M.

I am about to try fiddling with queue.mem.events ... but is there any general guidance about memory usage tuning for filebeat?

2

Well, setting queue.mem.events to 1024 doesn't seem to have been helpful:

(1) It seems to be struggling to fetch the backlog of data, and keep up with continuing log generation

(2) Filebeat pods are still running out of memory and crashing and restarting.

3

I've now found https://github.com/elastic/beats/issues/9302#issuecomment-490000600 which didn't exist last time I went through this loop (or, at least, the fix in 6.8.1 didn't exist). Trying it to see what happens ...

4

Hi @TimWard,

Yes, please, try with 6.8.1 if possible, we backported some fixes for memory leaks related to autodiscover to this version.

5

The removal of add_kubernetes_metadata made an enormous difference. The move to 6.8.1 may also have helped, but I did both together so am not sure. It's been running fine overnight, for the first time since we've tried running filebeat on Kubernetes.

Now, where did the add_kubernetes_metadata which caused the trouble come from? - I'm not absolutely certain but I think it was in the filebeat-k8s.yaml file which I downloaded from somewhere vaguely semi-official looking as "this is how you do filebeat on Kubernetes".

Here is the "after" memory usage on the same scale: image

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.