# Filebeat keeps open files forever **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [November 9, 2016, 7:19pm UTC](https://discuss.elastic.co/t/filebeat-keeps-open-files-forever/65548 "2016-11-09T19:19:02Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Srinivas\_Chamarthi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/srinivas_chamarthi/32/8729_2.png) [@Srinivas\_Chamarthi](https://discuss.elastic.co/u/Srinivas_Chamarthi)\ **Post date:** [November 9, 2016, 7:19pm UTC](https://discuss.elastic.co/t/filebeat-keeps-open-files-forever/65548/1 "2016-11-09T19:19:02Z")
Hi, I have a similar issue like [this](https://discuss.elastic.co/t/filebeat-keeps-files-open-forever/62802) and I have upgraded to latest version 5.0 to see if the issue goes away but I still see the same issue. below is the configuration and few debug lines where it mentions the file is ``` - paths: - /apps/opt/logs/*/*/*.log exclude_lines: ["^DEBUG"] input_type: log ignore_older: 1m close_inactive: 10s clean_removed: true clean_inactive: 10s close_removed: true close_renamed: true #force_close_files: true scan_frequency: 5s ``` 2016/11/08 23:49:43.090250 prospector\_log.go:269: DBG File rename was detected: /apps/opt/logs/application.log -\> /apps/opt/logs/application.23\_08Nov2016.34.log, Current offset: 21675185 2016/11/08 23:49:43.090275 prospector\_log.go:282: DBG File rename detected but harvester not finished yet. 2016/11/08 23:49:43.090304 prospector\_log.go:288: DBG Harvester for file is still running: /apps/opt/logs/application.23\_08Nov2016.34.log any help is appreciated. thx srinivas ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [November 10, 2016, 9:52am UTC](https://discuss.elastic.co/t/filebeat-keeps-open-files-forever/65548/2 "2016-11-10T09:52:42Z")
It looks like the indentation of your config file is off a little bit off for close\_removed and the options below. Can you correct this? `force_close_files` is not available anymore in 5.0. Are the files kept open "forever" or are they closed after `close_inactive`? How often do you update the files? ---
**Author:** ![nanshan](https://avatars.discourse-cdn.com/v4/letter/n/858c86/32.png) [@nanshan](https://discuss.elastic.co/u/nanshan)\ **Post date:** [November 11, 2016, 6:14pm UTC](https://discuss.elastic.co/t/filebeat-keeps-open-files-forever/65548/3 "2016-11-11T18:14:00Z")
Hi Ruflin, I am using filebeat 5.0, filebeat stills holds the deleted logs. I added in my file close\_inactive: 5m ---
**Author:** ![Srinivas\_Chamarthi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/srinivas_chamarthi/32/8729_2.png) [@Srinivas\_Chamarthi](https://discuss.elastic.co/u/Srinivas_Chamarthi)\ **Post date:** [November 11, 2016, 7:27pm UTC](https://discuss.elastic.co/t/filebeat-keeps-open-files-forever/65548/4 "2016-11-11T19:27:45Z")
Hi Ruflin, the files are open forever. The files get updated very often filling 750MB in less than 20 minutes or less. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [November 14, 2016, 1:58pm UTC](https://discuss.elastic.co/t/filebeat-keeps-open-files-forever/65548/5 "2016-11-14T13:58:00Z")
Can you guys share some log files in a gist? It is possible that the output did not catch up with reading? ---
**Author:** ![Srinivas\_Chamarthi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/srinivas_chamarthi/32/8729_2.png) [@Srinivas\_Chamarthi](https://discuss.elastic.co/u/Srinivas_Chamarthi)\ **Post date:** [November 16, 2016, 5:17am UTC](https://discuss.elastic.co/t/filebeat-keeps-open-files-forever/65548/6 "2016-11-16T05:17:28Z")
we have IOT process logs which are shared using NFS mounts from four vms. do you need any other information ? ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [November 16, 2016, 7:49am UTC](https://discuss.elastic.co/t/filebeat-keeps-open-files-forever/65548/7 "2016-11-16T07:49:35Z")
It would be nice to see some full log files from filebeat. You can share them in a gist. Be aware that in general it is not recommended to fetch log files from mounted volumes but have filebeat installed on all edge nodes. ---
**Author:** ![Srinivas\_Chamarthi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/srinivas_chamarthi/32/8729_2.png) [@Srinivas\_Chamarthi](https://discuss.elastic.co/u/Srinivas_Chamarthi)\ **Post date:** [December 2, 2016, 1:59am UTC](https://discuss.elastic.co/t/filebeat-keeps-open-files-forever/65548/8 "2016-12-02T01:59:18Z")
Hi Ruflin, apologize for late reply since I moved onto other stuff and with holidays. But I have to look into this one more final time. I think I see whats going on. Our process actually writes the log files and it rotates the log file once it reaches say 750MB in less than 10 minutes . then the filebeat reads the logs. I am using lsof -p on the filebeat and I notice that filebeat still holds a reference to the deleted file even though it doesn't exist and never closes . I am not sure if filebeat is done reading the file completely ( I am assuming its not). I am using close\_removed and clean\_removed attributes but still it doesn't look like filebeat is releasing the files ever. I am not sure how to debug this further. Please let me know if there is anything else I can do to avoid this situation. thx sri ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [December 5, 2016, 3:39pm UTC](https://discuss.elastic.co/t/filebeat-keeps-open-files-forever/65548/9 "2016-12-05T15:39:30Z")
Best is to have a look at the log lines on what it is stating there. If the harvester is still open / catching up, it explains why the files are still open. In case the output is not blocked, close\_removed should still apply as soon as the event is sent. But here it could be that the network drive comes into play and filebeat gets some cached data instead of being notified that it is removed. Note: I don't know the details of NFS mount implementation. Please have a look at the log files and let me know what you see there. Best with debug level then you should see what is happening (or not). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 2, 2017, 3:39pm UTC](https://discuss.elastic.co/t/filebeat-keeps-open-files-forever/65548/10 "2017-01-02T15:39:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.