# Auditbeat \[7.11.2 and 7.12.0\] memory issue **URL:** \ **Category:** Beats\ **Tags:** auditbeat\ **Created:** [March 30, 2021, 5:57pm UTC](https://discuss.elastic.co/t/auditbeat-7-11-2-and-7-12-0-memory-issue/268830 "2021-03-30T17:57:59Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![mareckii](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mareckii/32/86347_2.png) [@mareckii](https://discuss.elastic.co/u/mareckii)\ **Post date:** [March 30, 2021, 5:58pm UTC](https://discuss.elastic.co/t/auditbeat-7-11-2-and-7-12-0-memory-issue/268830/1 "2021-03-30T17:58:00Z")
Hi, I see memory issue when i'm using **add\_process\_metadata** processor. Memory usage is growing until OOM ![Screenshot 2021-03-30 at 08.15.33](https://us1.discourse-cdn.com/elastic/original/3X/b/9/b92db66d7200a2da8a62a87d2f17343a7a43138f.png) when i remove this processor from my config: ![Screenshot 2021-03-30 at 19.51.02](https://us1.discourse-cdn.com/elastic/original/3X/2/1/217f73bbb9664a68782dc53ce7f650429819cc24.png) I have dump, but I can't upload here. ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [March 31, 2021, 3:14pm UTC](https://discuss.elastic.co/t/auditbeat-7-11-2-and-7-12-0-memory-issue/268830/2 "2021-03-31T15:14:20Z")
Hi @mareckii, welcome to discuss 🙂 Could you share the configuration you are using? It'd be nice to have the profile, if we can confirm this is an bug maybe you can create an issue in github and upload the profile there. ---
**Author:** ![mareckii](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mareckii/32/86347_2.png) [@mareckii](https://discuss.elastic.co/u/mareckii)\ **Post date:** [March 31, 2021, 3:36pm UTC](https://discuss.elastic.co/t/auditbeat-7-11-2-and-7-12-0-memory-issue/268830/3 "2021-03-31T15:36:37Z")
Hi, I'm using configuration from your example here: [https://raw.githubusercontent.com/elastic/beats/7.12/deploy/kubernetes/auditbeat-kubernetes.yaml](https://raw.githubusercontent.com/elastic/beats/7.12/deploy/kubernetes/auditbeat-kubernetes.yaml) only difference is: > output.logstash: > hosts: ["127.0.0.1:5044"] currently i removed: > - add\_process\_metadata: > match\_pids: ['process.pid'] > include\_fields: ['container.id'] Just let me know where I'll upload profile ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [March 31, 2021, 4:52pm UTC](https://discuss.elastic.co/t/auditbeat-7-11-2-and-7-12-0-memory-issue/268830/4 "2021-03-31T16:52:35Z")
@mareckii I have been doing some quick tests with a simple auditbeat configuration and I have seen that the `add_process_metadata` processor has a [cache](https://github.com/elastic/beats/blob/6454736e68db1fedadedd9e137a480450def4181/libbeat/processors/add_process_metadata/cache.go#L38) for processes information whose entries are never released, but it only has an entry for each process id, so it is effectively limited by the maximum number of pids in the system. I have tried to create many processes in a loop, to fill this cache, and the memory usage of `add_process_metadata` seems to grow till about 13MB, but doesn't seem to go beyond that, so it seems to be effectively limited. ![Captura de pantalla de 2021-03-31 18-43-46](https://us1.discourse-cdn.com/elastic/original/3X/a/8/a830ae253718da9f726abbac9714dd1a22e32734.png) Even if it could be nice to remove entries of non-existing processes, 13MB doesn't seem so problematic. Could you check in your heap dump if the memory is being consumed by `add_process_metadata` or by `add_kubernetes_metadata`? I haven't tried with `add_kubernetes_metadata`, but with the configuration you are using, `add_kubernetes_metadata` won't enrich events if they don't have the `container.id`, so removing `add_process_metadata`, may also reduce the memory usage of the other processor. Could you try to remove `add_kubernetes_metadata` while keeping `add_process_metadata` and check if memory usage improves? ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [March 31, 2021, 4:58pm UTC](https://discuss.elastic.co/t/auditbeat-7-11-2-and-7-12-0-memory-issue/268830/5 "2021-03-31T16:58:32Z")
Looking to `add_kubernetes_metadata`, it also has a cache, that relies on receiving delete events from the Kubernetes API to delete its entries. The problem could be there, there are cases where delete events are not always received. These caches would stay in memory forever. **Update** : Pods are also removed from the cache if they are marked for termination in an update. In any case it'd be good to check if the memory issue is there. @mareckii it'd be great if you could confirm in your scenario if the problem is with `add_process_metadata` or with `add_kubernetes_metadata`. ---
**Author:** ![mareckii](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mareckii/32/86347_2.png) [@mareckii](https://discuss.elastic.co/u/mareckii)\ **Post date:** [March 31, 2021, 6:43pm UTC](https://discuss.elastic.co/t/auditbeat-7-11-2-and-7-12-0-memory-issue/268830/6 "2021-03-31T18:43:18Z")
This is how it looks in my dump ![Screenshot 2021-03-31 at 20.34.48](https://us1.discourse-cdn.com/elastic/original/3X/2/f/2fc05fda76dd7f02793adc4b3d78f19bbff493ee.png) There is nothing related with `add_kubernetes_metadata` on my dump. ---
**Author:** ![mareckii](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mareckii/32/86347_2.png) [@mareckii](https://discuss.elastic.co/u/mareckii)\ **Post date:** [April 1, 2021, 7:23am UTC](https://discuss.elastic.co/t/auditbeat-7-11-2-and-7-12-0-memory-issue/268830/7 "2021-04-01T07:23:24Z")
I disabled `add_kubernetes_metadata` and enabled `add_process_metadata ` memory usage looks like here: ![Screenshot 2021-04-01 at 08.16.51](https://us1.discourse-cdn.com/elastic/original/3X/4/6/46391f8b3fecfadf43e40c62d8b6423ce02a7ba1.png) When i disable `add_process_metadata` and enable `add_kubernetes_metadata` usage is here: ![Screenshot 2021-03-31 at 20.48.34](https://us1.discourse-cdn.com/elastic/original/3X/d/8/d83c2b53c4be72790b0a562008b0eaa50fd5dff3.png) I've tried to increase memory limit form 200M to 1G it takes more time (several days) but finally i got OOM ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [April 1, 2021, 9:38am UTC](https://discuss.elastic.co/t/auditbeat-7-11-2-and-7-12-0-memory-issue/268830/8 "2021-04-01T09:38:38Z")
Thanks @mareckii for continuing with the investigation. I have created an issue in github, could you please try to attach your heap dumps to a comment there? [Memory leak with add\_process\_metadata and k8s manifest for Auditbeat · Issue #24890 · elastic/beats · GitHub](https://github.com/elastic/beats/issues/24890) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 29, 2021, 11:39am UTC](https://discuss.elastic.co/t/auditbeat-7-11-2-and-7-12-0-memory-issue/268830/9 "2021-04-29T11:39:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.