Filebeat pods keeps increasing Memory usage

Elastic Stack Beats
,
8

Hello Andres,

I have been checking your dumps and I can see for now that two things are most memory consuming the registry reader (GetFields function) and the kubernetes_metadata library.
(I am not going to add any screenshots here, will only add more noise)

To our suggestions then:

  1. Try to remove the add_kubernetes_metatada config. The metadata enrichment should happen in the background. We try to avoid any double calls of the library.
  2. If first does not work try add_resource_metadata.namespace: false . I forgot to mention it. Let see if this can workaround your issue
  3. I see a lot of lines in your logs like the following:
{"log.level":"info","@timestamp":"2023-02-13T15:14:06.940Z","log.logger":"input.harvester","log.origin":{"file.name":"log/harvester.go","file.line":337},"message":"Reader was closed. Closing.","service.name":"filebeat","input_id":"e02069c0-1226-442e-afb0-f3d6cd1bb401","source_file":"/var/log/containers/ad-login-ingress-nginx-controller-6bcfd98bc8-k9jqs_ad-login_controller-56bc20f3e0b1e6f9ced94e04ed3406d2b9b8e23dfe440b75101b1bcf52fb3eba.log","state_id":"native::258154-2049","finished":false,"os_id":"258154-2049","old_source":"/var/log/containers/ad-login-ingress-nginx-controller-6bcfd98bc8-k9jqs_ad-login_controller-56bc20f3e0b1e6f9ced94e04ed3406d2b9b8e23dfe440b75101b1bcf52fb3eba.log","old_finished":true,"old_os_id":"258154-2049","harvester_id":"00bbbb86-a60b-4c81-937d-f40e60fb2a46","ecs.version":"1.6.0"}

Can you please describe me how you conduct the tests? Is it like you kill the pods and recreating them?

  1. I would also try to minimise the problem to see what condition can be problematic from what you have in your configuration. Can you just keep one condition at a time and repeat tests? Like that we could minimise and focus to the problematic case

We are investigating with the team and get back to you.