# \[SOLVED\] Filebeat keeps open files forever **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [January 26, 2016, 8:31am UTC](https://discuss.elastic.co/t/solved-filebeat-keeps-open-files-forever/40098 "2016-01-26T08:31:03Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![hamelg](https://avatars.discourse-cdn.com/v4/letter/h/ecae2f/32.png) [@hamelg](https://discuss.elastic.co/u/hamelg)\ **Post date:** [January 26, 2016, 8:31am UTC](https://discuss.elastic.co/t/solved-filebeat-keeps-open-files-forever/40098/1 "2016-01-26T08:31:03Z")
Here is our setup : ``` filebeat: prospectors: - paths: - "/var/log/apache2/*access.log" document_type: accesslog ignore_older: 5m tail_files: true ``` Every day, logrotate rotates these files. ``` /var/log/apache2/*.log { daily rotate 7 compress notifempty create 644 root adm sharedscripts postrotate if [-f '/var/run/apache2.pid']; then /etc/init.d/apache2 reload > /dev/null fi endscript } ``` Sometimes (pretty rarely : about once on ten), filebeat keeps old files opened forever. lsof shows us that filebeat is at EOF, but for some unknown reasons it doesn't close the file. Documentation says about force\_close\_files : > By default, Filebeat keeps the files that it’s reading open until the timespan specified by ignore\_older has elapsed. This behaviour can cause issues when a file is removed. is there a known issue about that ? What kinds of issue can happen ? If filebeat reaches EOF on a deleted file, why it keeps it opened ? I am worry about using force\_close\_files because I could lose some logs. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [January 26, 2016, 3:16pm UTC](https://discuss.elastic.co/t/solved-filebeat-keeps-open-files-forever/40098/2 "2016-01-26T15:16:25Z")
Filebeat should latest close a file after ignore\_older, in your case 5 minutes. With the next major release we will introduce close\_older as ignore\_older was having to different purposes which could have conflicted: [https://github.com/elastic/beats/pull/718](https://github.com/elastic/beats/pull/718) The file is kept open as filebeat doesn't know, if the file only was renamed and it will continue reading, if the file was actually deleted or it just disappeared for a moment. About force\_close\_files: The important part is in the second part of the documentation. If you use force\_close\_files rotated files are only picked up again after scan\_frequency. If the file was removed in the meantime and not all log lines were read before rotating, these log lines are lost. Which filebeat version are you using? ---
**Author:** ![hamelg](https://avatars.discourse-cdn.com/v4/letter/h/ecae2f/32.png) [@hamelg](https://discuss.elastic.co/u/hamelg)\ **Post date:** [January 26, 2016, 3:43pm UTC](https://discuss.elastic.co/t/solved-filebeat-keeps-open-files-forever/40098/3 "2016-01-26T15:43:38Z")
we use filebeat 1.0.1 I understand that we have no other choice to use force\_close\_files until v2.0 . am i correct ? thank you ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [January 27, 2016, 9:44am UTC](https://discuss.elastic.co/t/solved-filebeat-keeps-open-files-forever/40098/4 "2016-01-27T09:44:33Z")
As you use ignore\_older: 5m, is already "locking" the file for 5 minutes an issue? ---
**Author:** ![hamelg](https://avatars.discourse-cdn.com/v4/letter/h/ecae2f/32.png) [@hamelg](https://discuss.elastic.co/u/hamelg)\ **Post date:** [January 27, 2016, 3:06pm UTC](https://discuss.elastic.co/t/solved-filebeat-keeps-open-files-forever/40098/5 "2016-01-27T15:06:49Z")
I'm not sure to understand how it works. We have changed the setup, now we have : ``` filebeat: prospectors: - paths: - "/var/log/apache2/*.log" document_type: accesslog force_close_files: true tail_files: true ``` files are rotated at 06:00am : xxx.log is rename to xxx.log.1 xxx.log.1 is compressed to xxx.log.1.gz xxx.log.1 is deleted apache writes now logs to xxx.log Filebeat has still the previous log file open xxx.log.1 ??? why, it should be closed, because force\_close\_file is enabled ? I must wait 24H (default value for ignore\_older) before filebeat closes the deleted file ? # lsof |grep delete|grep beat .... filebeat 27131 root 20r REG 252,2 2906279 136754 /var/log/apache2/xxx.log.1 (deleted) in the syslog, we see these errors : Jan 27 08:13:29 /usr/bin/filebeat[27131]: log.go:94: File reading error. Stopping harvester. Error: Force closing file: /var/log/apache2/xxx.log ---
**Author:** ![hamelg](https://avatars.discourse-cdn.com/v4/letter/h/ecae2f/32.png) [@hamelg](https://discuss.elastic.co/u/hamelg)\ **Post date:** [January 28, 2016, 8:51am UTC](https://discuss.elastic.co/t/solved-filebeat-keeps-open-files-forever/40098/6 "2016-01-28T08:51:00Z")
After playing with ignore\_older and force\_close\_files, we notice some issues caused by file rotation : - filebeat lets files open forever. With the time, these files eat free space on the filesystem. - when ignore\_older timer has elapsed without modifications, filebeat ignores definitively the file . When this happen, we must restart filebeat. It seems these different timers management to poll file/directory modifications are subject to race condition issues. have you considered to use the inotify API to improve filebeat reliability ? At now, we are going to try increase scan\_frequency timer and see if it fixes our issues. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [January 28, 2016, 11:21am UTC](https://discuss.elastic.co/t/solved-filebeat-keeps-open-files-forever/40098/7 "2016-01-28T11:21:46Z")
@hamelg that's the reason close\_older was introduced and ignore\_older is set to infinity by default. See [post by @ruflin](https://discuss.elastic.co/t/filebeat-keeps-open-files-forever/40098/2) ---
**Author:** ![hamelg](https://avatars.discourse-cdn.com/v4/letter/h/ecae2f/32.png) [@hamelg](https://discuss.elastic.co/u/hamelg)\ **Post date:** [January 28, 2016, 3:02pm UTC](https://discuss.elastic.co/t/solved-filebeat-keeps-open-files-forever/40098/8 "2016-01-28T15:02:43Z")
So until v2.0, the workaround is to restart filebeat at regular time to free deleted files. I don't see other solution. How can I test close\_older feature, is there a v2.0 snapshot preview somewhere ? thank you ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [January 28, 2016, 4:53pm UTC](https://discuss.elastic.co/t/solved-filebeat-keeps-open-files-forever/40098/9 "2016-01-28T16:53:17Z")
you can try the nightlies: [https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/](https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/) ---
**Author:** ![hamelg](https://avatars.discourse-cdn.com/v4/letter/h/ecae2f/32.png) [@hamelg](https://discuss.elastic.co/u/hamelg)\ **Post date:** [January 29, 2016, 8:05am UTC](https://discuss.elastic.co/t/solved-filebeat-keeps-open-files-forever/40098/10 "2016-01-29T08:05:07Z")
close\_older has solved the issue 😌 when close\_older will be present, v1.2 or v2 ? Thank you ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [January 29, 2016, 8:21am UTC](https://discuss.elastic.co/t/solved-filebeat-keeps-open-files-forever/40098/11 "2016-01-29T08:21:25Z")
It will be for sure in 2.0. We are probably not going to backport it to 2.0. In case you run with the master build, please let us know in case you have any issues. ---
**Author:** ![iyaozhen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iyaozhen/32/7276_2.png) [@iyaozhen](https://discuss.elastic.co/u/iyaozhen)\ **Post date:** [March 29, 2016, 6:01am UTC](https://discuss.elastic.co/t/solved-filebeat-keeps-open-files-forever/40098/12 "2016-03-29T06:01:07Z")
I think it is maybe a bug. @ruflin My paths conf is `/home/work/IP/*/log/ModuleCall.log.*`. This 11 module create log very fast. Write file in current hour log, example `ModuleCall.log.2016032913`. And every hour del before day log example `ModuleCall.log.2016032813`, But filebeat still open this file: `lr-x------ 1 work work 64 3月 29 12:58 96 -> /home/work/IP/adapter/log/ModuleCall.log.2016032813 (deleted)` Disk space seemingly don't freed. Disk used fast grow to 100%. ![](https://us1.discourse-cdn.com/elastic/original/2X/b/bb40e174f351b522d50c41ff4bfe66021698fa77.png) My full configure: filebeat 1.1.2 ```auto paths: - /home/work/IP/*/log/ModuleCall.log.* encoding: utf-8 fields: app_name: channel_log fields_under_root: true ignore_older: 2h scan_frequency: 20s harvester_buffer_size: 65536 tail_files: false backoff: 3s force_close_files: true ``` It is a very big problem for me. When I kill filebeat: ![](https://us1.discourse-cdn.com/elastic/original/2X/1/120a0134c6c37cf3f4f785abdb9cf16c7ccb1b69.png) Please forgive my poor english. Thx everyone. ---
**Author:** ![mosheka](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mosheka/32/9754_2.png) [@mosheka](https://discuss.elastic.co/u/mosheka)\ **Post date:** [May 15, 2016, 1:35pm UTC](https://discuss.elastic.co/t/solved-filebeat-keeps-open-files-forever/40098/13 "2016-05-15T13:35:35Z")
Hi, We have filebeat installed on Amazon AMI w/ log rotated deployed and automatic deletion of old files. We faced the forvever opened and deleted files and tried to implement the close\_older as described in [https://www.elastic.co/guide/en/beats/filebeat/1.2/configuration-filebeat-options.html](https://www.elastic.co/guide/en/beats/filebeat/1.2/configuration-filebeat-options.html) (seems it was impelemented before 2.0) Yet, when we restarted the service, the configuration did not do much difference (in the end we implemented a cron w/ hourly restart) We'll be glad to know if there is more elegant way to implement it. filebeat --version filebeat version 1.2.2 (amd64) /etc/filebeat/filebeat.yml ```auto filebeat: prospectors: - paths: - /var/app/current/logs/*.log input_type: log document_type: app_log close_older: 1m registry_file: /var/lib/filebeat/registry output: logstash: hosts: ["xxx.xxx.xxx.xxx:5044"] shipper: logging: files: rotateeverybytes: 10485760 # = 10MB ``` ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [May 17, 2016, 6:28am UTC](https://discuss.elastic.co/t/solved-filebeat-keeps-open-files-forever/40098/14 "2016-05-17T06:28:35Z")
Your configuration looks good. Does it keep all files open or only some specific ones? ---
**Author:** ![mosheka](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mosheka/32/9754_2.png) [@mosheka](https://discuss.elastic.co/u/mosheka)\ **Post date:** [May 17, 2016, 6:59am UTC](https://discuss.elastic.co/t/solved-filebeat-keeps-open-files-forever/40098/15 "2016-05-17T06:59:15Z")
All the files (we did not keep it for a long time (we kept it for 15 min), but it opened all the log files, did not close any of them though only one file was active, and deleted files were remained opened. We restarted the daemon several times ---
**Author:** ![mohit.s](https://avatars.discourse-cdn.com/v4/letter/m/76d3ee/32.png) [@mohit.s](https://discuss.elastic.co/u/mohit.s)\ **Post date:** [May 18, 2016, 6:18am UTC](https://discuss.elastic.co/t/solved-filebeat-keeps-open-files-forever/40098/16 "2016-05-18T06:18:34Z")
Above resolution does not solve the issue. I am using latest version of filebeat. Yaml file on windows. **force\_close\_files: true** **ignore\_older: 1m** Files are not rotated. Everytime i have to restart FileBeat service. Any resolution ? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [May 18, 2016, 11:04am UTC](https://discuss.elastic.co/t/solved-filebeat-keeps-open-files-forever/40098/17 "2016-05-18T11:04:28Z")
see sample config given. Use `close_older`. No need to enbale `force_close_files` or set `ignore_older`. ---
**Author:** ![mohit.s](https://avatars.discourse-cdn.com/v4/letter/m/76d3ee/32.png) [@mohit.s](https://discuss.elastic.co/u/mohit.s)\ **Post date:** [May 18, 2016, 2:05pm UTC](https://discuss.elastic.co/t/solved-filebeat-keeps-open-files-forever/40098/18 "2016-05-18T14:05:10Z")
Thanks πŸ™‚ ---
**Author:** ![ori.rubinfeld](https://avatars.discourse-cdn.com/v4/letter/o/7c8e57/32.png) [@ori.rubinfeld](https://discuss.elastic.co/u/ori.rubinfeld)\ **Post date:** [July 28, 2016, 10:43am UTC](https://discuss.elastic.co/t/solved-filebeat-keeps-open-files-forever/40098/19 "2016-07-28T10:43:46Z")
There is a bug with filebeat 1.2.2, had the same issue. upgraded to filebeat 1.2.3 ---
**Author:** ![dwradcliffe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dwradcliffe/32/11764_2.png) [@dwradcliffe](https://discuss.elastic.co/u/dwradcliffe)\ **Post date:** [September 4, 2016, 9:00pm UTC](https://discuss.elastic.co/t/solved-filebeat-keeps-open-files-forever/40098/20 "2016-09-04T21:00:15Z")
Using `close_older` with 1.2.3 still isn't working for me. [Next page](https://discuss.elastic.co/t/solved-filebeat-keeps-open-files-forever/40098.md?page=2)