# Need a logical end of file definition for filebeat **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [February 5, 2016, 9:16pm UTC](https://discuss.elastic.co/t/need-a-logical-end-of-file-definition-for-filebeat/41099 "2016-02-05T21:16:45Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![tringuyen](https://avatars.discourse-cdn.com/v4/letter/t/a9a28c/32.png) [@tringuyen](https://discuss.elastic.co/u/tringuyen)\ **Post date:** [February 5, 2016, 9:16pm UTC](https://discuss.elastic.co/t/need-a-logical-end-of-file-definition-for-filebeat/41099/1 "2016-02-05T21:16:45Z")
I have problem with the log that defines end of file by a string, the end of file can be in the middle of the log, So filebeat needs to stop reading log at that line instead of reading all the way down to physical end-of-file. For example: 2016-02-04 11:42:06.279-05:00 i360imsa UUMSyncWebService Verbose 0 Successfully 2016-02-04 11:42:06.279-05:00 i360imsa UUMSyncWebService Verbose 0 Process Sync Message - 2016-02-04 11:42:06.328-05:00 i360imsa UUMSyncWebService Verbose 0 GetAllPBXIDDuplicates was called 2016-02-04 11:42:06.447-05:00 i360imsa UUMSyncWebService Verbose 0 Successfully returnning 2016-02-04 11:42:06.447-05:00 i360imsa UUMSyncWebService Informational 0 Process Sync Messages **Logger Information: EOF** 2016-02-01 11:42:06.447-05:00 i360imsa UUMSyncWebService Informational 0 Process Sync Messages **Logger Information: EOF** is end of log file. new lines of log will be inserted before this line. If the log file is full, it will overwrite the log from the beginning. Could you please help me workaround this case? currently, filebeat just reads all the lines and it cannot know the new lines are inserted in the middle of the log file. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [February 8, 2016, 2:37pm UTC](https://discuss.elastic.co/t/need-a-logical-end-of-file-definition-for-filebeat/41099/2 "2016-02-08T14:37:14Z")
Interesting. Is there some content after `Logger Information: EOF` in the file? Is this line overwritten when new lines are added and put at new end of file? Or there other `Logger Information` messages? This use-case is not supported by beats yet, but feel free to add an enhancement request with some more details to github.com/elastic/beats. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [February 9, 2016, 8:01am UTC](https://discuss.elastic.co/t/need-a-logical-end-of-file-definition-for-filebeat/41099/3 "2016-02-09T08:01:36Z")
Can you share some more information on what kind of logging system this is? ---
**Author:** ![tringuyen](https://avatars.discourse-cdn.com/v4/letter/t/a9a28c/32.png) [@tringuyen](https://discuss.elastic.co/u/tringuyen)\ **Post date:** [February 11, 2016, 3:15pm UTC](https://discuss.elastic.co/t/need-a-logical-end-of-file-definition-for-filebeat/41099/4 "2016-02-11T15:15:20Z")
Yes, there are bunch of lines after Logger Information: EOF in the file. when the new lines are added, it will move the **Logger Information: EOF** down. All new lines are inserted before this line. the lines after this line will be overwritten. When the file is full, reach the max (for example 200Mb), it will start over from the beginning of the file, but all the old lines still there after **Logger Information: EOF**. Simply says: it re-cycles the file, but not clear the content when the file is full. ---
**Author:** ![tringuyen](https://avatars.discourse-cdn.com/v4/letter/t/a9a28c/32.png) [@tringuyen](https://discuss.elastic.co/u/tringuyen)\ **Post date:** [February 11, 2016, 3:24pm UTC](https://discuss.elastic.co/t/need-a-logical-end-of-file-definition-for-filebeat/41099/5 "2016-02-11T15:24:32Z")
when the file is full and the line Logger Information: EOF is at the end of file, it will start over from the beginning of the file. So this line likes a marker to tell the logger that this is the point to write new log lines, not write at the physical end of file ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [February 12, 2016, 9:55am UTC](https://discuss.elastic.co/t/need-a-logical-end-of-file-definition-for-filebeat/41099/6 "2016-02-12T09:55:15Z")
@tringuyen It would be interesting to know what kind of application creates these kind of log files. It somehow sounds also going in the direction of unifiedbeat or circular log files: [https://github.com/cleesmith/unifiedbeat](https://github.com/cleesmith/unifiedbeat) ---
**Author:** ![tringuyen](https://avatars.discourse-cdn.com/v4/letter/t/a9a28c/32.png) [@tringuyen](https://discuss.elastic.co/u/tringuyen)\ **Post date:** [February 12, 2016, 3:07pm UTC](https://discuss.elastic.co/t/need-a-logical-end-of-file-definition-for-filebeat/41099/7 "2016-02-12T15:07:44Z")
Thank you for asking @ruflin. This is our own company logger framework. I looked at unifiedbeat, but it is not the same. it is circular log file. the system just creates 1 log file and circular it. This can be the algorithm: - Add a new attribute to yalm file, let say - EOF: a string. by default, it is nothing and filebeat will read the log to end of the file. -If it has a string defined for EOF, filebeat will read the file until it reaches this string. -Circular case: If it cannot find the string at the end of the file, it can go back to the beginning of the file and read until it reaches the string. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [February 12, 2016, 3:38pm UTC](https://discuss.elastic.co/t/need-a-logical-end-of-file-definition-for-filebeat/41099/8 "2016-02-12T15:38:35Z")
that' not enough + on filebeat restart we have to figure out where to continue from. As file may have rotated since last read and there are no meta-data in file to discover file being rotated + old offset is valid starting from X it would be hard create a simple generic + robust config dealing with this "fileformat". ---
**Author:** ![tringuyen](https://avatars.discourse-cdn.com/v4/letter/t/a9a28c/32.png) [@tringuyen](https://discuss.elastic.co/u/tringuyen)\ **Post date:** [February 12, 2016, 3:48pm UTC](https://discuss.elastic.co/t/need-a-logical-end-of-file-definition-for-filebeat/41099/9 "2016-02-12T15:48:08Z")
Thanks @steffens. Yes, you are right. we need to handle filebeat restarted case too. But I think this handle is the same for all kind of logs. The file is rotated when filebeat reads from the last point to the end of the file and it cannot see the EOF string. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [February 15, 2016, 8:47am UTC](https://discuss.elastic.co/t/need-a-logical-end-of-file-definition-for-filebeat/41099/10 "2016-02-15T08:47:29Z")
@tringuyen Are there any other log file reader that support this kind of format (including rotation). Would be perhaps interesting to have a look to see how they do it. ---
**Author:** ![tringuyen](https://avatars.discourse-cdn.com/v4/letter/t/a9a28c/32.png) [@tringuyen](https://discuss.elastic.co/u/tringuyen)\ **Post date:** [February 18, 2016, 2:05pm UTC](https://discuss.elastic.co/t/need-a-logical-end-of-file-definition-for-filebeat/41099/11 "2016-02-18T14:05:16Z")
unfortunately, I am still find it @ruflin. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [February 18, 2016, 4:14pm UTC](https://discuss.elastic.co/t/need-a-logical-end-of-file-definition-for-filebeat/41099/12 "2016-02-18T16:14:38Z")
Let me know if you find one that does. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [February 18, 2016, 5:10pm UTC](https://discuss.elastic.co/t/need-a-logical-end-of-file-definition-for-filebeat/41099/13 "2016-02-18T17:10:28Z")
Problem is, there is not enough meta-data in file to find a good restarting point. You basically have to scan your file for the EOF marker and check if old offset is still correct. If file was rotated more than once between filebeat restarts, offset is always wrong (but there is no way to detect this). ---
**Author:** ![tringuyen](https://avatars.discourse-cdn.com/v4/letter/t/a9a28c/32.png) [@tringuyen](https://discuss.elastic.co/u/tringuyen)\ **Post date:** [February 18, 2016, 5:16pm UTC](https://discuss.elastic.co/t/need-a-logical-end-of-file-definition-for-filebeat/41099/14 "2016-02-18T17:16:22Z")
when it restarts , it should read from the point it left. if more the once rotation, we lost data - no way for this situation. so the file should big enough and filebeat off-time should be short enough. Thank you so much for your all replies ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 9:55pm UTC](https://discuss.elastic.co/t/need-a-logical-end-of-file-definition-for-filebeat/41099/15 "2017-07-05T21:55:35Z")