# About clean\_removed in Filebeat 5.0.0-alpha5 **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [September 7, 2016, 8:34am UTC](https://discuss.elastic.co/t/about-clean-removed-in-filebeat-5-0-0-alpha5/59949 "2016-09-07T08:34:35Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![vin](https://avatars.discourse-cdn.com/v4/letter/v/a87d85/32.png) [@vin](https://discuss.elastic.co/u/vin)\ **Post date:** [September 7, 2016, 8:34am UTC](https://discuss.elastic.co/t/about-clean-removed-in-filebeat-5-0-0-alpha5/59949/1 "2016-09-07T08:34:35Z")
Filebeat config: > filebeat: > prospectors: > - > paths: > - /var/log/applog/app\_\*.log > input\_type: log > document\_type: applog > ignore\_older: 2m > close\_eof: true > clean\_inactive: 5m > close\_removed: true > clean\_removed: true After I created app\_0.log in /var/log/applog/ , I could see log: > 2016-09-07T19:52:18+08:00 INFO End of file reached: /var/log/applog/app\_0.log. Closing because close\_eof is enabled. Then I deleted app\_0.log, but app\_0.log was still in registry file. I guess it's affected by close\_eof config. So in this case how can I clean the state of removed file? And more, when I created many file like app\_0.log again and then deleted them , these ERR log appeared continuously: > 2016-09-07T20:08:48+08:00 ERR State for /var/log/applog/app\_0.log should have been dropped, but couldn't as state is not finished. And there's also log like this: > 2016-09-07T20:25:40+08:00 ERR File is falling under ignore\_older before harvesting is finished. Adjust your close\_\* settings Shouldn't these ERR logs stop printing again and again ? ---
**Author:** ![vin](https://avatars.discourse-cdn.com/v4/letter/v/a87d85/32.png) [@vin](https://discuss.elastic.co/u/vin)\ **Post date:** [September 7, 2016, 9:57am UTC](https://discuss.elastic.co/t/about-clean-removed-in-filebeat-5-0-0-alpha5/59949/2 "2016-09-07T09:57:50Z")
About this ERR log: > 2016-09-07T20:25:40+08:00 ERR File is falling under ignore\_older before harvesting is finished. Adjust your close\_\* settings I got your explain at [Filebeat Stop Cleaning Registry - #8 by Gabry993](https://discuss.elastic.co/t/filebeat-stop-cleaning-registry/58902/8) : > The first error is shown, in case a file did not finish harvesting yet but already falls under ignore\_older. This can for example happen when a file is not updated anymore, but the harvester can't finish reading it as the output is not available or too slow. As soon as the output catches up, the file will be closed and ignored. But in my case, the ERR log is appeared when start filebeat, the file should not begin harvesting, not "in case a file did not finish harvesting yet " . ---
**Author:** ![vin](https://avatars.discourse-cdn.com/v4/letter/v/a87d85/32.png) [@vin](https://discuss.elastic.co/u/vin)\ **Post date:** [September 7, 2016, 11:31am UTC](https://discuss.elastic.co/t/about-clean-removed-in-filebeat-5-0-0-alpha5/59949/3 "2016-09-07T11:31:46Z")
After I delete clean\_removed config from filebeat: > ## filebeat: prospectors: > > paths: > > - /var/log/applog/app\_\*.log > input\_type: log > document\_type: applog > ignore\_older: 2m > close\_eof: true > clean\_inactive: 5m clean\_inactive worked as expected, but when file state is cleaned from registry, these ERR log begin be printed continusly: > 2016-09-07T23:16:17+08:00 ERR File is falling under ignore\_older before harvesting is finished. Adjust your close\_\* settings: After I deleted the file which was already cleaned from registry, the ERR log never appeared. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [September 8, 2016, 7:02am UTC](https://discuss.elastic.co/t/about-clean-removed-in-filebeat-5-0-0-alpha5/59949/4 "2016-09-08T07:02:41Z")
Thanks for all the details. Some comments - ignore\_older applies to the modification date of the file. So if you file is 1:59 minutes last modified on startup, harvester will open it and start harvesting it, then ignore\_older is going to start complain the next second. As soon as close\_eof is reached, it should be ignored in the next scan - You mention that it should have never started harvesting. So on startup the file was already older then ignore\_older? Strangely on startup all files are (should) be set to finished. Can you share some more log outputs (all lines?) - What is the way your logs are rotated? - What do you mean by you deleted it from the registry? You manually edited the registry? - Could you check the nightly build if you still see the same issue? [https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/](https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/) If you find a way to reliably reproduce the above that would be great. ---
**Author:** ![vin](https://avatars.discourse-cdn.com/v4/letter/v/a87d85/32.png) [@vin](https://discuss.elastic.co/u/vin)\ **Post date:** [September 9, 2016, 6:45am UTC](https://discuss.elastic.co/t/about-clean-removed-in-filebeat-5-0-0-alpha5/59949/5 "2016-09-09T06:45:27Z")
•You mention that it should have never started harvesting. So on startup the file was already older then ignore\_older? **--yes**. Strangely on startup all files are (should) be set to finished. **--I have lots of history files, and I just need lastest 2h files.** Can you share some more log outputs (all lines?) --Filebeat log file was full of ERR log like this: > 2016-09-07T23:16:17+08:00 ERR File is falling under ignore\_older before harvesting is finished. Adjust your close\_\* settings: •What is the way your logs are rotated? --It's just lots of 10M size files. All the files are only written once and not updated from time to time. •What do you mean by you deleted it from the registry? You manually edited the registry? --I mean I deleted the log file, not deleted the state of file from registry. The problem I most want to solve is why so many ERR logs in /var/log/filebeat/filebeat and how should I avoid it? I'm re describing the problem : Filebeat Config: > ## filebeat: prospectors: > > paths: > > - /var/log/applog/app\_\*.log > input\_type: log > document\_type: applog > ignore\_older: 2h > close\_inactive: 5m > clean\_inactive: 4h Under /var/log/applog/ there were lots of history files, most were modified before 2h. And I deleted /var/lib/filebeat/registry before start filebeat. As a result, /var/log/filebeat/filebeat was full of these ERR logs: > 2016-09-07T23:16:17+08:00 ERR File is falling under ignore\_older before harvesting is finished. Adjust your close\_\* settings: ---
**Author:** ![dooblem](https://avatars.discourse-cdn.com/v4/letter/d/ba9def/32.png) [@dooblem](https://discuss.elastic.co/u/dooblem)\ **Post date:** [September 9, 2016, 12:58pm UTC](https://discuss.elastic.co/t/about-clean-removed-in-filebeat-5-0-0-alpha5/59949/6 "2016-09-09T12:58:04Z")
Hello, I'm getting the same Error, with a prospector with similar config: > paths: > - /var/www/magento/var/report/\* > input\_type: log > document\_type: magento\_report > fields: > environment: test > fields\_under\_root: true > ignore\_older: 24h > clean\_inactive: 25h Log sending seems to work fine though. just my filebeat.log being a few megas because of this error. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [September 12, 2016, 7:17am UTC](https://discuss.elastic.co/t/about-clean-removed-in-filebeat-5-0-0-alpha5/59949/7 "2016-09-12T07:17:08Z")
The error both of you mentioned happens in the following case: - Prospector or harvester detected that a file meets one of the `close_*` criterias and marks the state to be removed - Registrar tries to remove the state but harvester is still running, so state is not removed as otherwise the file would be started reading again from the beginning @vin Do you see this problem only when you startup with old files and the errors disappear after some time or the errors are constant? I'm thinking if perhaps at the first run filebeat somehow opens also ignore\_older files somehow (which should not be the case). I'm currently investigating this. @dooblem What is the exact version you are using? Can you also share some more details on your log rotation? ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [September 12, 2016, 7:44am UTC](https://discuss.elastic.co/t/about-clean-removed-in-filebeat-5-0-0-alpha5/59949/8 "2016-09-12T07:44:20Z")
As a heads up: I think I managed to reproduce the issue. So far it seems like mainly a logging issue but I need to do some more investigations. I keep you posted. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [September 12, 2016, 7:53am UTC](https://discuss.elastic.co/t/about-clean-removed-in-filebeat-5-0-0-alpha5/59949/9 "2016-09-12T07:53:40Z")
Here is the PR to fix this issue: [https://github.com/elastic/beats/pull/2517](https://github.com/elastic/beats/pull/2517) It seems like the issue was only related to logging. Thanks a lot for helping to find this issue. I will ping you as soon as a snapshot build with the fix is available. @vin In your first post you also mentioned you have issue with the following error: ```auto ERR State for /var/log/applog/app_0.log should have been dropped, but couldn't as state is not finished. ``` Is this error also printed again and again or does it disappear after the first time? ---
**Author:** ![dooblem](https://avatars.discourse-cdn.com/v4/letter/d/ba9def/32.png) [@dooblem](https://discuss.elastic.co/u/dooblem)\ **Post date:** [September 12, 2016, 8:41am UTC](https://discuss.elastic.co/t/about-clean-removed-in-filebeat-5-0-0-alpha5/59949/10 "2016-09-12T08:41:45Z")
Hello @ruflin . Thanks a lot. I' using Filebeat 5.0.0-alpha5. Remember [Can Filebeat watch for new files created in a directory?](https://discuss.elastic.co/t/can-filebeat-watch-for-new-files-created-in-a-directory/57811) 🙂 ? Log files are created but not removed. I just want old files to be ignored. Let me know if you need more info. It will be hard for me to test, unless you release another alpha6 .deb... Thanks in advance ! ---
**Author:** ![vin](https://avatars.discourse-cdn.com/v4/letter/v/a87d85/32.png) [@vin](https://discuss.elastic.co/u/vin)\ **Post date:** [September 12, 2016, 10:44am UTC](https://discuss.elastic.co/t/about-clean-removed-in-filebeat-5-0-0-alpha5/59949/11 "2016-09-12T10:44:23Z")
@ruflin I reproduced the issue I mentioned in my first post, it happend in this situation: 1. The state of app\_0.log and app\_1.log was exist in registry file, but app\_0.log was deleted.(Indeed I upgraded filebeat from 1.2.3) 2. I started filebeat and these ERR logs came out: (which u already confirmed as a logging issue) > ERR File is falling under ignore\_older before harvesting is finished. Adjust your close\_\* settings: 1. With the config clean\_removed: true , the state of removed file was still in registry. 2. And after clean\_inactive(which is 5m), the state of inactive file was still in registry. 3. Then I created a file app\_2.log , these logs occured: > INFO Harvester started for file: /var/log/app\_2.log > ... > ERR State for /var/log/app\_0.log should have been dropped, but couldn't as state is not finished. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [September 12, 2016, 2:06pm UTC](https://discuss.elastic.co/t/about-clean-removed-in-filebeat-5-0-0-alpha5/59949/12 "2016-09-12T14:06:04Z")
Fix should be available in the next hours in the snapshot build as it got merged: [https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/](https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/) @dooblem No chance to remember all names with filebeat versions 😉 ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [September 12, 2016, 2:08pm UTC](https://discuss.elastic.co/t/about-clean-removed-in-filebeat-5-0-0-alpha5/59949/13 "2016-09-12T14:08:30Z")
@vin Can you try to do the same again with filebeat 5.0 only from the latest snapshot to see if this still happens? ---
**Author:** ![vin](https://avatars.discourse-cdn.com/v4/letter/v/a87d85/32.png) [@vin](https://discuss.elastic.co/u/vin)\ **Post date:** [September 13, 2016, 1:32am UTC](https://discuss.elastic.co/t/about-clean-removed-in-filebeat-5-0-0-alpha5/59949/14 "2016-09-13T01:32:24Z")
@ruflin It's not easy for me to do the test with the nightly build filebeat, I can't open the url from my domain network. sorry... ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [September 13, 2016, 7:10am UTC](https://discuss.elastic.co/t/about-clean-removed-in-filebeat-5-0-0-alpha5/59949/15 "2016-09-13T07:10:07Z")
@vin I see. Alternative would be to build it from source yourself but that would require a Golang environment? Or you wait until we push beta1? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 28, 2016, 8:34am UTC](https://discuss.elastic.co/t/about-clean-removed-in-filebeat-5-0-0-alpha5/59949/16 "2016-09-28T08:34:35Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.