# Filebeat using accessive amount of memory on server **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [July 8, 2019, 3:30pm UTC](https://discuss.elastic.co/t/filebeat-using-accessive-amount-of-memory-on-server/189374 "2019-07-08T15:30:17Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![urvi](https://avatars.discourse-cdn.com/v4/letter/u/57b2e6/32.png) [@urvi](https://discuss.elastic.co/u/urvi)\ **Post date:** [July 8, 2019, 3:30pm UTC](https://discuss.elastic.co/t/filebeat-using-accessive-amount-of-memory-on-server/189374/1 "2019-07-08T15:30:17Z")
Hello Team, I am using fileebat 6.5.1. We have server that produce large volume of data per day(75 gb). I am facing memory issue on this server for filebeat. Filebeat is keep on increasing memory usage every day almost 7GB. can you please suggest some configuration to overcome this issue. This is my input configuration: #=========================== Filebeat inputs ============================= filebeat.inputs: # Each - is an input. Most options can be set at the input level, so # you can use different inputs for various configurations. # Below are the input specific configurations. - type: log ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [July 9, 2019, 12:36am UTC](https://discuss.elastic.co/t/filebeat-using-accessive-amount-of-memory-on-server/189374/2 "2019-07-09T00:36:06Z")
Do you process a large number of small files per day, or a small number of large files per day? Memory leaks in on harvester startup/shutdown have been resolved in filebeat 6.8.1. ---
**Author:** ![urvi](https://avatars.discourse-cdn.com/v4/letter/u/57b2e6/32.png) [@urvi](https://discuss.elastic.co/u/urvi)\ **Post date:** [July 9, 2019, 8:34am UTC](https://discuss.elastic.co/t/filebeat-using-accessive-amount-of-memory-on-server/189374/3 "2019-07-09T08:34:06Z")
there are number of process running on that servers. Each process keep on creating log file on average say in every hour by log rotation. For example when file is 5MB it will create rotation files. Almost producing 70 GB data per day. registerty file size is almost 14MB. I have started filebeat yesterday (started with memory utilization 163MB)and today it reach upto 5703 MB. If this continue we will face disk running out of space. Is the configuration is correct ? Can you suggest proper configuration to handle this issue. Is there no other way other then moving to 6.8.1 ? ---
**Author:** ![urvi](https://avatars.discourse-cdn.com/v4/letter/u/57b2e6/32.png) [@urvi](https://discuss.elastic.co/u/urvi)\ **Post date:** [July 9, 2019, 8:46am UTC](https://discuss.elastic.co/t/filebeat-using-accessive-amount-of-memory-on-server/189374/4 "2019-07-09T08:46:13Z")
> [@urvi](#): > > close\_timeout: 5m Can you please also advise that close\_timeout: 5m is the correct configuration ? as per his description on fileebeat docs it will close the file handler after given interval irrespective of the reading position in file. We have applied this setting but still facing memory issue. Does this means it is not closing handler properly ? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [July 10, 2019, 1:09pm UTC](https://discuss.elastic.co/t/filebeat-using-accessive-amount-of-memory-on-server/189374/5 "2019-07-10T13:09:38Z")
This scenario sounds like you have a many many small files. About 1400 files per day. This is quite a lot. Is there a particular reason to rotate at 5MB? How about switching to 1GB (have 70 files per day)? The fixes in 6.8.1 should fix a memory leak exactly in this scenario: many many small files with rather short processing time. For how long to you keep these files around? How many files do you have on disk? 14MB registry file is hughe. Sounds like loads of CPU time is spend when the registry file snapshot is written. Does the regsitry file hold files not on disk anymore (registry file is in JSON, you can easily parse it)? A low `close_timeout` is ok. If the file is still on disk and the offset does not point to the end of file, then filebeat will pick it up again and continue from last known position. The memory leak found in filebeat is triggered by closing files often, though. Best way to mitigate this is: reduce number of files to process in total and make sure we don't need to close early. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 7, 2019, 1:09pm UTC](https://discuss.elastic.co/t/filebeat-using-accessive-amount-of-memory-on-server/189374/6 "2019-08-07T13:09:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.