# Potential memory leak issue with filebeat and metricbeat **URL:** \ **Category:** Beats\ **Tags:** filebeat, metricbeat\ **Created:** [May 25, 2023, 4:37pm UTC](https://discuss.elastic.co/t/potential-memory-leak-issue-with-filebeat-and-metricbeat/334353 "2023-05-25T16:37:57Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![kbujold\_wr](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@kbujold\_wr](https://discuss.elastic.co/u/kbujold_wr)\ **Post date:** [May 25, 2023, 4:37pm UTC](https://discuss.elastic.co/t/potential-memory-leak-issue-with-filebeat-and-metricbeat/334353/1 "2023-05-25T16:37:57Z")
Since upgrading from ELK 7.17.1 to ELK 8.6.2 (and even with ELK 8.7.1) we are experiencing OOMKilled on filebeat and metricbeat pods. We had no issues with ELK 7.17.1. Increasing the resources allocations does not resolve the issue and simply delays the crash. This appears to be a memory leak issue with beats. ```auto State: Running Started: Thu, 25 May 2023 15:18:43 +0000 Last State: Terminated Reason: OOMKilled Exit Code: 137 Started: Thu, 25 May 2023 02:53:22 +0000 Finished: Thu, 25 May 2023 15:18:41 +0000 Ready: True ``` This is an example of our filebeat pod memory in the past 24 hours ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/a/4ac3c1192edb9208c4de898c1897fb45c17c9631.png) We have tried this config which was mentioned in other posts but it makes no differences. We also do not use cron jobs. ```auto processors: - add_kubernetes_metadata: add_resource_metadata: deployment: false cronjob: false ``` ---
**Author:** ![kbujold\_wr](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@kbujold\_wr](https://discuss.elastic.co/u/kbujold_wr)\ **Post date:** [May 30, 2023, 1:36pm UTC](https://discuss.elastic.co/t/potential-memory-leak-issue-with-filebeat-and-metricbeat/334353/2 "2023-05-30T13:36:18Z")
We are also seeing this in ELK 8.8.0 ---
**Author:** ![cmacknz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cmacknz/32/103773_2.png) [@cmacknz](https://discuss.elastic.co/u/cmacknz)\ **Post date:** [May 30, 2023, 7:30pm UTC](https://discuss.elastic.co/t/potential-memory-leak-issue-with-filebeat-and-metricbeat/334353/3 "2023-05-30T19:30:24Z")
Can you post your complete Filebeat configuration? What we need is heap profiles from Filebeat which should tell us what is using the memory. The instructions to do this are: 1. Start the Beat process with `httpprof` (profiling) enabled. This allows us to easily extract memory profiles of the running process. Add these configuration options: ```auto http.host: localhost http.port: 6060 http.pprof.enabled: true ``` 1. Once Beats is started and is done initializing (after 5-10 minutes), you can collect the first memory dump via a simple curl command like this: `curl -s -v http://localhost:8080/debug/pprof/heap > heap_normal.bin`. 2. Once you start noticing that the process is taking excessive amounts of memory, a second dump needs to be generated like `curl -s -v http://localhost:8080/debug/pprof/heap > heap_high.bin`. If you attach the .bin files we can analyze them to see what is going on. The profile taken when the memory usage is excessive is the most important one. ---
**Author:** ![kbujold\_wr](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@kbujold\_wr](https://discuss.elastic.co/u/kbujold_wr)\ **Post date:** [June 14, 2023, 7:55pm UTC](https://discuss.elastic.co/t/potential-memory-leak-issue-with-filebeat-and-metricbeat/334353/4 "2023-06-14T19:55:33Z")
I am unable to reply with the .bin attachments. How would you like me to share? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [June 14, 2023, 7:56pm UTC](https://discuss.elastic.co/t/potential-memory-leak-issue-with-filebeat-and-metricbeat/334353/5 "2023-06-14T19:56:58Z")
@kbujold_wr You can use [pastebin](https://pastebin.com/) ---
**Author:** ![kbujold\_wr](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@kbujold\_wr](https://discuss.elastic.co/u/kbujold_wr)\ **Post date:** [June 14, 2023, 8:28pm UTC](https://discuss.elastic.co/t/potential-memory-leak-issue-with-filebeat-and-metricbeat/334353/6 "2023-06-14T20:28:05Z")
pastebin is blocked at my workplace. Can you access here? [https://drive.google.com/drive/folders/1K-gtu\_mcpLw6M3ocg8nDbtk0j8SkJBVs?usp=sharing](https://drive.google.com/drive/folders/1K-gtu_mcpLw6M3ocg8nDbtk0j8SkJBVs?usp=sharing) Below is our config map for filebeat. Note we have the same issue with metricbeat. ```auto Name: mon-filebeat-daemonset-config Namespace: monitor Labels: app=mon-filebeat app.kubernetes.io/managed-by=Helm chart=filebeat-8.5.1 helm.toolkit.fluxcd.io/name=filebeat helm.toolkit.fluxcd.io/namespace=monitor heritage=Helm release=mon-filebeat Annotations: meta.helm.sh/release-name: mon-filebeat meta.helm.sh/release-namespace: monitor remoteconfigchecksum: 12341234abcdabcd Data ==== filebeat.yml: ---- fields: system: name: yow-cgcs-supermicro-2 uid: d1374af9-1e53-4339-92be-974f1b033347 fields_under_root: true filebeat.autodiscover: providers: - hints.default_config: close_renamed: true paths: - /var/log/containers/*-${data.kubernetes.container.id}.log type: container hints.enabled: true host: ${NODE_NAME} type: kubernetes filebeat.inputs: - close_timeout: 5m enabled: true exclude_files: - ^/var/log/containers/ - ^/var/log/pods/ paths: - /var/log/*.log - /var/log/messages - /var/log/syslog - /var/log/**/*.log type: log http.port: 5066 monitoring: cluster_uuid: ${CLUSTER_UUID} elasticsearch: hosts: - https://mon-elasticsearch-client:9200 password: ${beats_system_monitoring_password} ssl.certificate_authorities: - /usr/share/filebeat/ext-ca.crt username: ${beats_system_monitoring_user} enabled: ${BEAT_MONITORING_ENABLED} name: ${NODE_NAME} output.elasticsearch: enabled: false host: ${NODE_NAME} hosts: - https://mon-elasticsearch-client:9200 ilm.pattern: "000001" index: ${INDEX_NAME}-%{+yyyy.MM.dd} password: ${ELASTICSEARCH_PASSWORD} protocol: https ssl.certificate_authorities: - /usr/share/filebeat/ext-ca.crt username: ${ELASTICSEARCH_USERNAME} output.file: enabled: false output.logstash: enabled: true hosts: - mon-logstash:5044 ssl.certificate: /usr/share/filebeat/config/instance/filebeat.crt ssl.certificate_authorities: - /usr/share/filebeat/ca.crt - /usr/share/filebeat/previous/ca.crt - /usr/share/filebeat/next/ca.crt ssl.key: /usr/share/filebeat/config/instance/filebeat.key timeout: 9 processors: - add_kubernetes_metadata: annotations.dedot: true default_matchers.enabled: false labels.dedot: true setup.dashboards: enabled: false setup.kibana: host: mon-kibana:5601 password: ${filebeat_password} protocol: https ssl.certificate_authorities: - /usr/share/filebeat/ext-ca.crt ssl.verification_mode: none username: ${filebeat_user} setup.template: name: ${INDEX_NAME} pattern: ${INDEX_PATTERN} BinaryData ==== Events: ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [June 14, 2023, 11:46pm UTC](https://discuss.elastic.co/t/potential-memory-leak-issue-with-filebeat-and-metricbeat/334353/7 "2023-06-14T23:46:56Z")
@cmacknz ^^^ ---
**Author:** ![cmacknz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cmacknz/32/103773_2.png) [@cmacknz](https://discuss.elastic.co/u/cmacknz)\ **Post date:** [June 15, 2023, 9:06pm UTC](https://discuss.elastic.co/t/potential-memory-leak-issue-with-filebeat-and-metricbeat/334353/8 "2023-06-15T21:06:18Z")
Here's what I see in the profiles, looking specifically at the heap in use space profile This is the normal case, nothing seems like it dominates it: ![Screen Shot 2023-06-15 at 4.59.38 PM](https://us1.discourse-cdn.com/elastic/original/3X/0/2/02112a14bfe1689d0706108045fc902b56895424.jpeg) Here's the case where the heap is high, the heap is dominated by allocations from the Kubernetes watch API. This could mean a few things, one is that it could be receiving very large responses from the Kubernetes API, but it could also mean that we are subscribing to too many events and the rate of allocations is too high. ![Screen Shot 2023-06-15 at 4.59.19 PM](https://us1.discourse-cdn.com/elastic/original/3X/1/e/1e36cfd25be36284c3cce0471090f76e7ba82301.png) This isn't something I have an immediate answer for. It seems similar to other cases I have seen before but it looks like we are still working on a fix. This is the first report, the entire discussion does have some comments with possible work arounds: [[Metricbeat] Possible memory leak with autodiscover · Issue #33307 · elastic/beats · GitHub](https://github.com/elastic/beats/issues/33307#issuecomment-1298222058) This is the follow up issue, which is scoped specifically to CronJobs: [`add_resource_metadata.cronjob` overloads the memory usage · Issue #31 · elastic/elastic-agent-autodiscover · GitHub](https://github.com/elastic/elastic-agent-autodiscover/issues/31) The interesting thing here is that CronJobs aren't involved at all. I'll ping the people working on those issues internally to see what they think. ---
**Author:** ![kbujold\_wr](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@kbujold\_wr](https://discuss.elastic.co/u/kbujold_wr)\ **Post date:** [June 27, 2023, 4:01pm UTC](https://discuss.elastic.co/t/potential-memory-leak-issue-with-filebeat-and-metricbeat/334353/9 "2023-06-27T16:01:27Z")
I have raised issue [Potential memory leak issue with filebeat and metricbeat · Issue #35796 · elastic/beats · GitHub](https://github.com/elastic/beats/issues/35796#) The problem still has not been found. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 25, 2023, 6:01pm UTC](https://discuss.elastic.co/t/potential-memory-leak-issue-with-filebeat-and-metricbeat/334353/10 "2023-07-25T18:01:52Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.