# FileBeat file is falling under ignore older **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [September 22, 2016, 8:16am UTC](https://discuss.elastic.co/t/filebeat-file-is-falling-under-ignore-older/61210 "2016-09-22T08:16:34Z")\ **Posts on this page:** 11\ **Page:** 2
**Author:** ![il.bert](https://avatars.discourse-cdn.com/v4/letter/i/ac91a4/32.png) [@il.bert](https://discuss.elastic.co/u/il.bert)\ **Post date:** [October 24, 2016, 10:40am UTC](https://discuss.elastic.co/t/filebeat-file-is-falling-under-ignore-older/61210/21 "2016-10-24T10:40:38Z")
Yes it updated to date of today Is filebeat relying on modification time of the file (windows) or to its internal mod time when computing the ignore\_older? I do really think this could be the point: as you say it is not reliable in windows and in my case I found many file with a modtime older than it's real content ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [October 24, 2016, 1:22pm UTC](https://discuss.elastic.co/t/filebeat-file-is-falling-under-ignore-older/61210/22 "2016-10-24T13:22:23Z")
For ignore\_older, it relies on the file mod\_time. For all the close\_\* config options it relies on the internal state. The issue with the mod time on Windows is the reason we switched from comparing the mod time to comparing the size when reading a file. So in case a file gets additional content but modtime is not updated, the new content will still be harvested by filebeat. I was hoping we would not face this issue with `ignore_older` as I kind of assumed the updated meta data is only flushed with a delay, but it seems like it is only flushed when closing the file. There is not really a solution we can do for other logs that are written, but I think we should at least have a solution for our own log file to make sure meta data is updated frequently (like every 5 minutes). I need to dig deeper here on what kind of write options exist here for Windows. @andrewkroh Any ideas? ---
**Author:** ![il.bert](https://avatars.discourse-cdn.com/v4/letter/i/ac91a4/32.png) [@il.bert](https://discuss.elastic.co/u/il.bert)\ **Post date:** [October 24, 2016, 2:17pm UTC](https://discuss.elastic.co/t/filebeat-file-is-falling-under-ignore-older/61210/23 "2016-10-24T14:17:07Z")
that's interesting but it seems kind of a bug to me why using two metrics? what about using the same metrics for close and ignore older? it seems to me that having 2 different metrics will always be a problem: ignore\_older must always be greater than close\_\* otherwise an error happen, but they may not be comparable! ---
**Author:** ![il.bert](https://avatars.discourse-cdn.com/v4/letter/i/ac91a4/32.png) [@il.bert](https://discuss.elastic.co/u/il.bert)\ **Post date:** [October 24, 2016, 2:34pm UTC](https://discuss.elastic.co/t/filebeat-file-is-falling-under-ignore-older/61210/24 "2016-10-24T14:34:34Z")
filebeat file has been created on the 21st as soon as filebeat.1 has been closed as you can see from log it is growing (last line at 16.30) but no modified date change ![](https://us1.discourse-cdn.com/elastic/original/2X/c/c125ba015e967ca8d50c25ef97cd18b5af1da2aa.png) ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [October 24, 2016, 2:47pm UTC](https://discuss.elastic.co/t/filebeat-file-is-falling-under-ignore-older/61210/25 "2016-10-24T14:47:46Z")
The difference is on purpose. All the `close_*` options are checked when the file is open and need the state information for comparison. The `ignore_older` option happens before a file handler is opened and does not rely on the internal state. `ignore_older` can also be applied, if there is no existing state, means no state has to be stored for old files. Normally `ignore_older` should be a magnitude larger then `close_inactive` which is the case for you so this error will never show up. Unfortunately there are these issue with Windows. ---
**Author:** ![il.bert](https://avatars.discourse-cdn.com/v4/letter/i/ac91a4/32.png) [@il.bert](https://discuss.elastic.co/u/il.bert)\ **Post date:** [October 24, 2016, 2:59pm UTC](https://discuss.elastic.co/t/filebeat-file-is-falling-under-ignore-older/61210/26 "2016-10-24T14:59:25Z")
what about check them both? ``` if windows_mod_date older_than_value AND ( !valid(last_internal_update) OR last_internal_update older_than_value ) then ignore ``` seems to be a reasonable condition if no internal state than skip that check if one of them is recent than keep it it both of them are older than value than throw error right now FB is clearly saying a wrong thing: my close\_\* setting are correct but it is anyway doing a comparison on two value that are not comparable it is clear FB cannot ignore a file that is beeing written right now ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [October 25, 2016, 7:31am UTC](https://discuss.elastic.co/t/filebeat-file-is-falling-under-ignore-older/61210/27 "2016-10-25T07:31:07Z")
I see your point that the error message can be confusing. The problem with the above suggestion is, that people will configure ingore\_older to be 24h and will start complain, if it doesn't get ignored after 24h. Now there is at least an error message that tells them that there is something "wrong". The reason we log an error is that I think this is also kind of a "bug" on the windows file system. Modtime should be when a file was modified, not closed after modified. Obviously there will be lots of people disagreeing here 🙂 I'm thinking of making it an Info message instead of an Err message. This makes it more obvious that in most cases no direct actions are needed. ---
**Author:** ![il.bert](https://avatars.discourse-cdn.com/v4/letter/i/ac91a4/32.png) [@il.bert](https://discuss.elastic.co/u/il.bert)\ **Post date:** [October 25, 2016, 8:28am UTC](https://discuss.elastic.co/t/filebeat-file-is-falling-under-ignore-older/61210/28 "2016-10-25T08:28:41Z")
I don't think people will complain: you are just overcoming a bug of OS (maybe only in windows) with a correct solution: this would provide the expected file close and ignore after all 🙂 if a file contains logs of now I don't want it to be ignored at all or I will loose lines. The important thing is to say that ignore\_older relies on the last written line not on the OS time. nobody will expect to ignore files that are being written now, I would complain if you try to ignore because I may be loosing lines just assume I have a close policy of 5 mins and ignore of 24h on bugged file with no moddate update: in case logger stops writing lines may be ignored after 5 mins of waiting, thus loosing next lines ---
**Author:** ![il.bert](https://avatars.discourse-cdn.com/v4/letter/i/ac91a4/32.png) [@il.bert](https://discuss.elastic.co/u/il.bert)\ **Post date:** [October 25, 2016, 8:47am UTC](https://discuss.elastic.co/t/filebeat-file-is-falling-under-ignore-older/61210/29 "2016-10-25T08:47:03Z")
However I think it is a good practice for any software to be as agnostic and independent as possible from external environment: especially when it is known to be full of errors (without doing names 😄 ) As a software developer I don't want that an error on external environment creates an error on my one, I feel responsible for what my software does 🙂 ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [October 31, 2016, 5:16pm UTC](https://discuss.elastic.co/t/filebeat-file-is-falling-under-ignore-older/61210/30 "2016-10-31T17:16:44Z")
Sorry for the late answer. It would be great, if we could figure out when the last line was written. Unfortunately the internal state we are having is, when the last line was read by us. One first step I did for future release is to already "downgrade" the message from Error to Info as otherwise it raises expectations that something is really wrong. But this is not the case, as in the end filebeat is doing "the right thing" and finishes reading the file. I will definitively keep an eye on this one to see if we can find some improvements, so we can rely on the timestamp from windows and are not required to do some additional windows magic (we already have enough of it in filebeat 😉 ). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 9:50pm UTC](https://discuss.elastic.co/t/filebeat-file-is-falling-under-ignore-older/61210/31 "2017-07-05T21:50:21Z")
Closed on July 5, 2017, 9:50pm UTC [Previous page](https://discuss.elastic.co/t/filebeat-file-is-falling-under-ignore-older/61210.md?page=1)