# \[Auditbeat\] Memory leak in 7.5.2 **URL:** \ **Category:** Beats\ **Created:** [February 7, 2020, 11:41am UTC](https://discuss.elastic.co/t/auditbeat-memory-leak-in-7-5-2/218335 "2020-02-07T11:41:04Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![nickbabkin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nickbabkin/32/57493_2.png) [@nickbabkin](https://discuss.elastic.co/u/nickbabkin)\ **Post date:** [February 7, 2020, 11:41am UTC](https://discuss.elastic.co/t/auditbeat-memory-leak-in-7-5-2/218335/1 "2020-02-07T11:41:04Z")
Hi! It seems like there's a memory leak in latest version of auditbeat (7.5.2) that is running with socket module enabled on high network loaded servers (such as load balancers). We recently upgraded to the newest version (7.5.2) from 6.8.0 (which had no similar issues) and started to experience intense memory loads. Not only memory usage was above what we think is normal (around 300-400 MB for a beat), it continued to grow up indefinitely until it went OOM and even crashed some of our servers. This only applies to our load balancers, memory usage on other type of servers seems to stay within the reasonable limits, although it might be due to the fact that there are not many active sockets. Some technical information: Affected version of auditbeat: 7.5.2 Aftected servers operating system: Ubuntu 16.04.5 LTS Auditbeat config: ```auto auditbeat.modules: - module: auditd # Load audit rules from separate files. Same format as audit.rules(7). audit_rule_files: ['${path.config}/audit.rules.d/*.conf'] - module: file_integrity paths: - /bin - /usr/bin - /sbin - /usr/sbin - /etc - /root/.ssh/authorized_keys - module: system datasets: - host # General host information, e.g. uptime, IPs - login # User logins, logouts, and system boots. - package # Installed, updated, and removed packages - process # Started and stopped processes - socket # Opened and closed sockets - user # User information # How often datasets send state updates with the # current state of the system (e.g. all currently # running processes, all open sockets). state.period: 1h # How often auditbeat queries for new processes, sockets etc metrics.period: 3s # Enabled by default. Auditbeat will read password fields in # /etc/passwd and /etc/shadow and store a hash locally to # detect any changes. user.detect_password_changes: true # File patterns of the login record files. login.wtmp_file_pattern: /var/log/wtmp* login.btmp_file_pattern: /var/log/btmp* output.logstash: hosts: *** ssl.certificate_authorities: *** bulk_max_size: 2096 timeout: 15 setup.template.name: "logstash_auditbeat_template" setup.template.pattern: "logstash-auditbeat-*" setup.template.settings: index.number_of_shards: 3 index.refresh_interval: 30s processors: - add_host_metadata: netinfo.enabled: true cache.ttl: 5m ``` We have the same configuration on all servers where auditbeat is deployed. Also attaching screenshots with memory usage on load balancer and other example server over last couple days (sudden drop of memory usage on load balancer was due to restarting auditbeat daemon). As you can see, it looks pretty okay on the server that doesn't have a load balancer role. ![Auditbeat Memory Usage - load balancer](https://us1.discourse-cdn.com/elastic/original/3X/3/3/3339ea89bd67304b49cdc0b564ce98b2d9fc80ea.png) ![Auditheat memory usage - other server](https://us1.discourse-cdn.com/elastic/original/3X/f/8/f88950012eeaa8004dc0ee106e1510bae6cb57d3.png) ---
**Author:** ![nickbabkin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nickbabkin/32/57493_2.png) [@nickbabkin](https://discuss.elastic.co/u/nickbabkin)\ **Post date:** [February 7, 2020, 11:44am UTC](https://discuss.elastic.co/t/auditbeat-memory-leak-in-7-5-2/218335/2 "2020-02-07T11:44:24Z")
Attaching latest profiler outputs from one of the affected servers. Socket module is definitely in the tops for memory usage and it grows endlessly. ``` go tool pprof http://localhost:8888/debug/pprof/heap Fetching profile over HTTP from http://localhost:8888/debug/pprof/heap Saved profile in /home/user/pprof/pprof.auditbeat.alloc_objects.alloc_space.inuse_objects.inuse_space.021.pb.gz File: auditbeat Build ID: b9be0e911c5db76838425b6aa234b98f3defe260 Type: inuse_space Time: Feb 7, 2020 at 12:38pm (CET) Entering interactive mode (type "help" for commands, "o" for options) (pprof) top Showing nodes accounting for 87.94MB, 100% of 87.94MB total Showing top 10 nodes out of 68 flat flat% sum% cum cum% 83.25MB 94.66% 94.66% 83.25MB 94.66% github.com/elastic/beats/x-pack/auditbeat/module/system/socket.(*state).createFlow 1MB 1.14% 95.80% 1MB 1.14% bytes.makeSlice 1MB 1.14% 96.94% 1MB 1.14% crypto/x509.parseCertificate 0.60MB 0.68% 97.62% 0.60MB 0.68% github.com/elastic/beats/auditbeat/module/auditd.(*MetricSet).receiveEvents 0.58MB 0.66% 98.28% 0.58MB 0.66% github.com/elastic/beats/x-pack/auditbeat/module/system/socket.(*dnsTracker).AddTransaction 0.51MB 0.58% 98.86% 0.51MB 0.58% regexp/syntax.(*compiler).inst 0.50MB 0.57% 99.43% 0.50MB 0.57% github.com/elastic/beats/vendor/github.com/json-iterator/go.init.4 0.50MB 0.57% 100% 0.50MB 0.57% github.com/elastic/beats/libbeat/outputs/kafka.init.ializers 0 0% 100% 1MB 1.14% bytes.(*Buffer).Write 0 0% 100% 1MB 1.14% bytes.(*Buffer).grow ``` ---
**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\ **Post date:** [February 7, 2020, 3:03pm UTC](https://discuss.elastic.co/t/auditbeat-memory-leak-in-7-5-2/218335/3 "2020-02-07T15:03:30Z")
Hi, can you share a profile created with (`--memprofile file`) and logs with `-d socket` (but not `-d '*'`) I'm curious about the lines containing `state flows=NNN [...]` in particular. Seems that flows are created but not expired for some reason. Also, does do you see it grow also on the other systems? Maybe just slowly so it doesn't become a problem so soon. ---
**Author:** ![nickbabkin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nickbabkin/32/57493_2.png) [@nickbabkin](https://discuss.elastic.co/u/nickbabkin)\ **Post date:** [February 10, 2020, 10:21am UTC](https://discuss.elastic.co/t/auditbeat-memory-leak-in-7-5-2/218335/4 "2020-02-10T10:21:15Z")
@adrisr, Attaching new profile as requested (couldn't find the way to attach files here so uploaded via firefox send): [link](https://send.firefox.com/download/2ce4ee128dde35e6/#Rx6trKsUHkEl-3E7CdQoAQ) Please let me know if any other information is required. Meanwhile, mem usage on the host reached 660 Mb/2Gb total and keep growing. There's a definite leak. > [@adrisr](#): > > Also, does do you see it grow also on the other systems? Maybe just slowly so it doesn't become a problem so soon. Not quite, others are stable, but they have significantly lower number of sockets as well. ---
**Author:** ![nickbabkin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nickbabkin/32/57493_2.png) [@nickbabkin](https://discuss.elastic.co/u/nickbabkin)\ **Post date:** [February 11, 2020, 10:01am UTC](https://discuss.elastic.co/t/auditbeat-memory-leak-in-7-5-2/218335/5 "2020-02-11T10:01:57Z")
@adrisr You were asking for data from other hosts (that are not load balancers). As you can see on the screenshot, memory looks stable during last 72 hours. ![Screenshot 2020-02-11 at 11.00.22 AM](https://us1.discourse-cdn.com/elastic/original/3X/4/3/43a25c7794b4583efd2f1639262675c353c2bc3a.png) Shall we create a github issue for a confirmed bug, or you can not confirm it yet? ---
**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\ **Post date:** [February 12, 2020, 8:38am UTC](https://discuss.elastic.co/t/auditbeat-memory-leak-in-7-5-2/218335/6 "2020-02-12T08:38:37Z")
@nickbabkin unfortunately there is not a lot of information in this profile. It only accounts for 3MB of allocated objects. I understand you got it via the `-memprof` argument which would have written the profile when the beat is terminated and most memory has been freed. Can you try running Auditbeat with `-httpprof :8888` (or any other port number) and then wait until the memory it's high and fetch a profile using: > curl '[http://localhost:8888/debug/pprof/heap?debug=1&gc](http://localhost:8888/debug/pprof/heap?debug=1&gc)' -o mem\_profile.txt > curl '[http://localhost:8888/debug/pprof/goroutine?debug=1](http://localhost:8888/debug/pprof/goroutine?debug=1)' -o goroutines.txt Also, did you have a chance to grep Auditbeat logs for the lines containing `state flows=` ? ---
**Author:** ![nickbabkin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nickbabkin/32/57493_2.png) [@nickbabkin](https://discuss.elastic.co/u/nickbabkin)\ **Post date:** [February 18, 2020, 9:38am UTC](https://discuss.elastic.co/t/auditbeat-memory-leak-in-7-5-2/218335/7 "2020-02-18T09:38:42Z")
Hi @adrisr, Here are the latest memprofile and goroutines after we ran the beat for 3 days: > **[Firefox Send](https://send.firefox.com/download/1b920de6bc9508e4/#5z9HwMTme-k8g37vRA8XLw)** > > Encrypt and send files with a link that automatically expires to ensure your important documents don’t stay online forever. As per sockets, is there any way to send this file privately to you? It has some sensitive data inside, and we have an NDA with Elastic. Thank you. ---
**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\ **Post date:** [February 18, 2020, 9:58am UTC](https://discuss.elastic.co/t/auditbeat-memory-leak-in-7-5-2/218335/8 "2020-02-18T09:58:07Z")
Sure, just send me a private message here or an email to adrian at elastic co ---
**Author:** ![nickbabkin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nickbabkin/32/57493_2.png) [@nickbabkin](https://discuss.elastic.co/u/nickbabkin)\ **Post date:** [February 18, 2020, 10:08am UTC](https://discuss.elastic.co/t/auditbeat-memory-leak-in-7-5-2/218335/9 "2020-02-18T10:08:11Z")
Sockets file is here: > **[Firefox Send](https://send.firefox.com/download/5b0c0199bdcbd83b/#0t1LHS__SLyMSBEa622Krw)** > > Encrypt and send files with a link that automatically expires to ensure your important documents don’t stay online forever. Password is sent to your email. Thanks! ---
**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\ **Post date:** [February 18, 2020, 2:02pm UTC](https://discuss.elastic.co/t/auditbeat-memory-leak-in-7-5-2/218335/10 "2020-02-18T14:02:24Z")
From the logs it's clear that the number of monitored sockets is increasing constantly, going from about 400 to 64k in 5 days. This means either there is this many sockets open in your system (some app is creating and never closing them) or that Auditbeat is missing the `inet_release` events that cause it to cleanup sockets, about 8 events of this type lost per minute, but I don't see that many lost events in the logs. Can you compare the number of open sockets in the system vs Auditbeat's state? > $ grep 'state flows=' logs/auditbeat | tail -1 | grep -o 'sockets=[0-9]\*' vs > $ sudo ss -pntu | wc -l ---
**Author:** ![nickbabkin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nickbabkin/32/57493_2.png) [@nickbabkin](https://discuss.elastic.co/u/nickbabkin)\ **Post date:** [February 18, 2020, 4:16pm UTC](https://discuss.elastic.co/t/auditbeat-memory-leak-in-7-5-2/218335/11 "2020-02-18T16:16:30Z")
@adrisr, I think we now have a clear indication that something is very wrong in the code: ``` live2:root@balancer:~# grep 'state flows=' /var/log/syslog* | tail -1 | grep -o 'sockets=[0-9]*' sockets=64275 live2:root@balancer:~# ss -pntu | wc -l 553 ``` ---
**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\ **Post date:** [February 18, 2020, 4:30pm UTC](https://discuss.elastic.co/t/auditbeat-memory-leak-in-7-5-2/218335/12 "2020-02-18T16:30:26Z")
Thanks @nickbabkin, I can implement some expiration for sockets (as there is already for flows), so that they are cleaned up after some time. But I'd really would like to understand first what's going on, it still doesn't look like it's losing the socket close events. Can you share with me some logs with the `socketdetailed` selector? It needs to be enabled separatedly because it's really really verbose. Either `logging.selectors: ["*", "socketdetailed"]` or the `-d '*,socketdetailed'` command-line flag. It'll log an extra 15-20k lines per second on your system. That will help see which events is Auditbeat receiving and in which order. Try to have at least 5min worth of logs please. Thanks again for your collaboration. ---
**Author:** ![nickbabkin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nickbabkin/32/57493_2.png) [@nickbabkin](https://discuss.elastic.co/u/nickbabkin)\ **Post date:** [February 20, 2020, 10:28am UTC](https://discuss.elastic.co/t/auditbeat-memory-leak-in-7-5-2/218335/13 "2020-02-20T10:28:41Z")
HI @adrisr, Here's the information that you requested (beware, it's more than 1Gb when unarchived): > **[Firefox Send](https://send.firefox.com/download/e803f6bea332c443/#7KT0BsSqpMMP_VafMHgEXA)** > > Encrypt and send files with a link that automatically expires to ensure your important documents don’t stay online forever. Password is the same as for the previous file, you can find it in the email. Thanks and looking forward to the fix. ---
**Author:** ![nickbabkin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nickbabkin/32/57493_2.png) [@nickbabkin](https://discuss.elastic.co/u/nickbabkin)\ **Post date:** [February 25, 2020, 10:24am UTC](https://discuss.elastic.co/t/auditbeat-memory-leak-in-7-5-2/218335/14 "2020-02-25T10:24:01Z")
Hi @adrisr, Any updates here, were you able to find a reason for the leak? Thank you! ---
**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\ **Post date:** [February 27, 2020, 10:00am UTC](https://discuss.elastic.co/t/auditbeat-memory-leak-in-7-5-2/218335/15 "2020-02-27T10:00:45Z")
Hi @nickbabkin, I will prepare a fix soon when I find some time. Will you be able to test it? ---
**Author:** ![nickbabkin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nickbabkin/32/57493_2.png) [@nickbabkin](https://discuss.elastic.co/u/nickbabkin)\ **Post date:** [February 27, 2020, 10:21am UTC](https://discuss.elastic.co/t/auditbeat-memory-leak-in-7-5-2/218335/16 "2020-02-27T10:21:03Z")
Hi @adrisr, Yes, we'll try to help with testing. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 26, 2020, 12:21pm UTC](https://discuss.elastic.co/t/auditbeat-memory-leak-in-7-5-2/218335/17 "2020-03-26T12:21:11Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. ---
**Author:** ![xeraa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xeraa/32/48181_2.png) [@xeraa](https://discuss.elastic.co/u/xeraa)\ **Post date:** [April 6, 2020, 4:51pm UTC](https://discuss.elastic.co/t/auditbeat-memory-leak-in-7-5-2/218335/18 "2020-04-06T16:51:20Z")
---
**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\ **Post date:** [April 6, 2020, 4:52pm UTC](https://discuss.elastic.co/t/auditbeat-memory-leak-in-7-5-2/218335/19 "2020-04-06T16:52:41Z")
We've prepared a patched version of 7.6.2 for testing. Can you give it a try? Here's the packages: [https://drive.google.com/drive/folders/1r7nH1v7Oc3e2BRsJZ6Wuuz69gfa2DLwh?usp=sharing](https://drive.google.com/drive/folders/1r7nH1v7Oc3e2BRsJZ6Wuuz69gfa2DLwh?usp=sharing) And here's the PR with the changes: [https://github.com/elastic/beats/pull/17500](https://github.com/elastic/beats/pull/17500) ---
**Author:** ![nickbabkin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nickbabkin/32/57493_2.png) [@nickbabkin](https://discuss.elastic.co/u/nickbabkin)\ **Post date:** [April 14, 2020, 8:32am UTC](https://discuss.elastic.co/t/auditbeat-memory-leak-in-7-5-2/218335/20 "2020-04-14T08:32:38Z")
Hi @adrisr, I gave it to our DevOps guys and will share the details as soon as I have them. Thanks for the fix. [Next page](https://discuss.elastic.co/t/auditbeat-memory-leak-in-7-5-2/218335.md?page=2)