# Filebeat memory leak associated with failure to open harvesters **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [March 8, 2018, 2:53pm UTC](https://discuss.elastic.co/t/filebeat-memory-leak-associated-with-failure-to-open-harvesters/123100 "2018-03-08T14:53:03Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![jeffkirk1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeffkirk1/32/8080_2.png) [@jeffkirk1](https://discuss.elastic.co/u/jeffkirk1)\ **Post date:** [March 8, 2018, 2:53pm UTC](https://discuss.elastic.co/t/filebeat-memory-leak-associated-with-failure-to-open-harvesters/123100/1 "2018-03-08T14:53:03Z")
Hi folks, We recently began having problem with runaway memory leaks in Filebeats (observed on versions 6.1.3 and above) on various versions of Scientific Linux 6.x and CentOS 7.x. It appears to be associated with a permissions-denied error when trying to read user ~/.bash\_history files from an NFS auto mount volume. I saw many errors of this type in the filebeat log file on the affected machines: `2018-03-08T06:11:58.464-0800 ERROR log/prospector.go:459 Harvester could not be started on existing file: /home/norman/.bash_history, Err: Error setting up harvester: Harvester setup failed. Unexpected file opening error: Failed opening /home/norman/.bash_history: open /home/norman/.bash_history: permission denied` The filebeat.yml harvester config was ``` #-------------------------- USER LOGIN SHELL HISTORY ------------------------- - type: log paths: - /root/.*history - /home/*/.*history tags: ["history"] tail_files: true ``` After disabling the /home/\*/.\*history path and restarting Filebeats, the memory usage no longer appears to be climbing and the error messages in the filebeat log file stopped. This is not the only path defined in my config that includes wildcards, but it IS the only path defined on an NFS auto-mount volume. Filebeat is running as root so it doesn't have any problems reading the shell history files if the home directory is actually mounted, but it doesn't handle the dismounting of the user's home directory properly and starts leaking RAM. It does not recover by itself and eventually the affected machines start swapping. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [March 12, 2018, 9:20am UTC](https://discuss.elastic.co/t/filebeat-memory-leak-associated-with-failure-to-open-harvesters/123100/2 "2018-03-12T09:20:16Z")
Could you share your full config file? Any chance to test this with the most recent 6.2 release? BTW thanks a lot for all the details so far. It's great to have that kind of post which should make finding the issue on our end easier. ---
**Author:** ![jeffkirk1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeffkirk1/32/8080_2.png) [@jeffkirk1](https://discuss.elastic.co/u/jeffkirk1)\ **Post date:** [March 22, 2018, 7:13pm UTC](https://discuss.elastic.co/t/filebeat-memory-leak-associated-with-failure-to-open-harvesters/123100/3 "2018-03-22T19:13:13Z")
Hi Ruflin, I can't share the entire config file due to some proprietary concerns about the nature of our logging, which I don't want to reveal, but I can confirm that the problem persists with Elastic 6.2, Logstash 6.2.2, Filebeat 6.2.2, and it seems to only happen with NFS auto mounted home directories. We've decided not to try to log things this way in the future so this issue isn't uppermost on my list anymore. Jeff ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [March 26, 2018, 9:16am UTC](https://discuss.elastic.co/t/filebeat-memory-leak-associated-with-failure-to-open-harvesters/123100/4 "2018-03-26T09:16:45Z")
In general we recommend not to use Filebeat on mounted volumes as it can have some side effects: [https://www.elastic.co/guide/en/beats/filebeat/current/faq.html#filebeat-network-volumes](https://www.elastic.co/guide/en/beats/filebeat/current/faq.html#filebeat-network-volumes) I would still hope we don't leak memory. ---
**Author:** ![jeffkirk1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeffkirk1/32/8080_2.png) [@jeffkirk1](https://discuss.elastic.co/u/jeffkirk1)\ **Post date:** [March 27, 2018, 5:01pm UTC](https://discuss.elastic.co/t/filebeat-memory-leak-associated-with-failure-to-open-harvesters/123100/5 "2018-03-27T17:01:52Z")
I think you mean network-mounted volumes, not just mounted volumes, because not being able to use Beats on mounted volumes would kinda suck. 😉 But it does leak memory; it's 100% reproducible in our environment, regardless of OS (from various Scientific Linux 6.x distros through CentOS 7.4). ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [March 28, 2018, 8:55am UTC](https://discuss.elastic.co/t/filebeat-memory-leak-associated-with-failure-to-open-harvesters/123100/6 "2018-03-28T08:55:32Z")
Haha, yes definitively network-mounted volumes 🙂 Do you have any suspicious errors in your log file related to file opening or closing? What are the close options you use? Do you see any file which filebeat keeps open which it shouldn't? ---
**Author:** ![jeffkirk1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeffkirk1/32/8080_2.png) [@jeffkirk1](https://discuss.elastic.co/u/jeffkirk1)\ **Post date:** [April 2, 2018, 8:07pm UTC](https://discuss.elastic.co/t/filebeat-memory-leak-associated-with-failure-to-open-harvesters/123100/7 "2018-04-02T20:07:59Z")
Unfortunately I had to remove the logging for the files in the NFS-mounted directories and no longer have any example data to look at. However, the filebeat config didn't have any special directives for closing or opening files, just whatever settings filebeat uses by default. Since I have a functional workaround I don't need to proceed with this investigation any further for my own benefit. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [April 4, 2018, 6:03am UTC](https://discuss.elastic.co/t/filebeat-memory-leak-associated-with-failure-to-open-harvesters/123100/8 "2018-04-04T06:03:45Z")
@jeffkirk1 Thanks for getting back to me. Fully understand that there is not a lot of benefit for you to further investigate and I'm happy you have a working version. I will keep an eye on this issue as it might pop up in other places. ---
**Author:** ![jdonahue](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jdonahue/32/29654_2.png) [@jdonahue](https://discuss.elastic.co/u/jdonahue)\ **Post date:** [April 5, 2018, 8:07pm UTC](https://discuss.elastic.co/t/filebeat-memory-leak-associated-with-failure-to-open-harvesters/123100/9 "2018-04-05T20:07:34Z")
Hi @ruflin, we are also experiencing this memory leak due to failure to setup harvesters (Filebeat 6.2.2). In our case they are local files, so this is not isolated to NFS-mounted directories. From what I can tell, this comes down to [channel.SubOutlet](https://github.com/elastic/beats/blob/6.2/filebeat/prospector/log/prospector.go#L577) being created for the new harvester, but not cleaned up properly when `h.Setup()` throws an exception. I _think_ this is because something that happens in `harvester.Run()` is what actually cleans up the outlet. Another possibility is that the new harvester is getting added to a state or registry structure somewhere and forgotten, so garbage collection isn't reaping it. Here's some output from pprof. I would attach the pprof file itself, but the web interface here won't let me upload anything that isn't an image. If there's a way to send you that file, please let me know. ``` (pprof) top10 33159.58kB of 33159.58kB total ( 100%) Dropped 308 nodes (cum <= 165.80kB) Showing top 10 nodes out of 51 (cum >= 512.04kB) flat flat% sum% cum cum% 19975.92kB 60.24% 60.24% 19975.92kB 60.24% runtime.malg 7680.66kB 23.16% 83.40% 7680.66kB 23.16% github.com/elastic/beats/filebeat/channel.SubOutlet 2048.19kB 6.18% 89.58% 2048.19kB 6.18% github.com/elastic/beats/filebeat/prospector/log.NewHarvester 1357.91kB 4.10% 93.68% 1357.91kB 4.10% runtime.allgadd 1024.08kB 3.09% 96.76% 1024.08kB 3.09% runtime.acquireSudog 544.67kB 1.64% 98.41% 544.67kB 1.64% github.com/elastic/beats/libbeat/publisher/queue/memqueue.NewBroker 528.17kB 1.59% 100% 528.17kB 1.59% regexp.(*bitState).reset 0 0% 100% 528.17kB 1.59% github.com/elastic/beats/filebeat/beater.(*Filebeat).Run 0 0% 100% 512.04kB 1.54% github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1 0 0% 100% 512.04kB 1.54% github.com/elastic/beats/filebeat/channel.SubOutlet.func1 (pprof) list SubOutlet Total: 32.38MB ROUTINE ======================== github.com/elastic/beats/filebeat/channel.SubOutlet in /home/jeremy/src/go/src/github.com/elastic/beats/filebeat/channel/util.go 7.50MB 7.50MB (flat, cum) 23.16% of Total . . 15:// SubOutlet create a sub-outlet, which can be closed individually, without closing the . . 16:// underlying outlet. . . 17:func SubOutlet(out Outleter) Outleter { . . 18: s := &subOutlet{ . . 19: isOpen: atomic.MakeBool(true), 1MB 1MB 20: done: make(chan struct{}), 2MB 2MB 21: ch: make(chan *util.Data), 4.50MB 4.50MB 22: res: make(chan bool, 1), . . 23: } . . 24: . . 25: go func() { . . 26: for event := range s.ch { . . 27: s.res <- out.OnEvent(event) ``` ![profile001](https://us1.discourse-cdn.com/elastic/original/3X/0/a/0a33c9c7dffb271434bd84905fc5dab524e35fb3.png) Please let me know if there are more details I can provide, and I'm happy to test any potential solutions when we narrow down the problem. Thanks, Jeremy ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [April 6, 2018, 12:25pm UTC](https://discuss.elastic.co/t/filebeat-memory-leak-associated-with-failure-to-open-harvesters/123100/10 "2018-04-06T12:25:55Z")
@jdonahue Thanks so much for all the details and the detailed investigation. There seems to be indeed something potentially wrong with the SubOutlet. Could you open a Github issue on copy paste this over? [https://github.com/elastic/beats](https://github.com/elastic/beats) I'm hoping there you can attach any kind of files 🤞 Could you share in the Github issue also your config and exact OS? ---
**Author:** ![jdonahue](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jdonahue/32/29654_2.png) [@jdonahue](https://discuss.elastic.co/u/jdonahue)\ **Post date:** [April 6, 2018, 8:18pm UTC](https://discuss.elastic.co/t/filebeat-memory-leak-associated-with-failure-to-open-harvesters/123100/11 "2018-04-06T20:18:55Z")
@ruflin You're welcome! I created a github issue, and was able to attach the full pprof file: [https://github.com/elastic/beats/issues/6797](https://github.com/elastic/beats/issues/6797) Please let me know if there's anything else I can do, and thanks for looking into this. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [April 9, 2018, 5:54am UTC](https://discuss.elastic.co/t/filebeat-memory-leak-associated-with-failure-to-open-harvesters/123100/12 "2018-04-09T05:54:12Z")
Thanks, lets continue the discussion there. ---
**Author:** ![jdonahue](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jdonahue/32/29654_2.png) [@jdonahue](https://discuss.elastic.co/u/jdonahue)\ **Post date:** [April 13, 2018, 8:25pm UTC](https://discuss.elastic.co/t/filebeat-memory-leak-associated-with-failure-to-open-harvesters/123100/13 "2018-04-13T20:25:44Z")
@ruflin I see the fix for this issue has been merged, which is awesome! Thanks! Do you know which release version will contain this fix? ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [April 16, 2018, 6:47am UTC](https://discuss.elastic.co/t/filebeat-memory-leak-associated-with-failure-to-open-harvesters/123100/14 "2018-04-16T06:47:48Z")
It will for sure go into 6.3 ---
**Author:** ![jdonahue](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jdonahue/32/29654_2.png) [@jdonahue](https://discuss.elastic.co/u/jdonahue)\ **Post date:** [April 16, 2018, 6:05pm UTC](https://discuss.elastic.co/t/filebeat-memory-leak-associated-with-failure-to-open-harvesters/123100/15 "2018-04-16T18:05:42Z")
Fantastic, thanks again! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 14, 2018, 6:06pm UTC](https://discuss.elastic.co/t/filebeat-memory-leak-associated-with-failure-to-open-harvesters/123100/16 "2018-05-14T18:06:01Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.