Since upgrading from ELK 7.17.1 to ELK 8.6.2 (and even with ELK 8.7.1) we are experiencing OOMKilled on filebeat and metricbeat pods. We had no issues with ELK 7.17.1. Increasing the resources allocations does not resolve the issue and simply delays the crash. This appears to be a memory leak issue with beats.
Started: Thu, 25 May 2023 15:18:43 +0000
Last State: Terminated
Exit Code: 137
Started: Thu, 25 May 2023 02:53:22 +0000
Finished: Thu, 25 May 2023 15:18:41 +0000
This is an example of our filebeat pod memory in the past 24 hours
We have tried this config which was mentioned in other posts but it makes no differences. We also do not use cron jobs.
We are also seeing this in ELK 8.8.0
Can you post your complete Filebeat configuration?
What we need is heap profiles from Filebeat which should tell us what is using the memory. The instructions to do this are:
- Start the Beat process with
httpprof (profiling) enabled. This allows us to easily extract memory profiles of the running process. Add these configuration options:
- Once Beats is started and is done initializing (after 5-10 minutes), you can collect the first memory dump via a simple curl command like this:
curl -s -v http://localhost:8080/debug/pprof/heap > heap_normal.bin.
- Once you start noticing that the process is taking excessive amounts of memory, a second dump needs to be generated like
curl -s -v http://localhost:8080/debug/pprof/heap > heap_high.bin.
If you attach the .bin files we can analyze them to see what is going on. The profile taken when the memory usage is excessive is the most important one.
I am unable to reply with the .bin attachments. How would you like me to share?
pastebin is blocked at my workplace. Can you access here?
Below is our config map for filebeat. Note we have the same issue with metricbeat.
Annotations: meta.helm.sh/release-name: mon-filebeat
- close_timeout: 5m
Here's what I see in the profiles, looking specifically at the heap in use space profile
This is the normal case, nothing seems like it dominates it:
Here's the case where the heap is high, the heap is dominated by allocations from the Kubernetes watch API. This could mean a few things, one is that it could be receiving very large responses from the Kubernetes API, but it could also mean that we are subscribing to too many events and the rate of allocations is too high.
This isn't something I have an immediate answer for. It seems similar to other cases I have seen before but it looks like we are still working on a fix.
This is the first report, the entire discussion does have some comments with possible work arounds: [Metricbeat] Possible memory leak with autodiscover · Issue #33307 · elastic/beats · GitHub
This is the follow up issue, which is scoped specifically to CronJobs: `add_resource_metadata.cronjob` overloads the memory usage · Issue #31 · elastic/elastic-agent-autodiscover · GitHub
The interesting thing here is that CronJobs aren't involved at all. I'll ping the people working on those issues internally to see what they think.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.