# Filebeat - Saving what files Filebeat has Processed in ES

**URL:** <https://discuss.elastic.co/t/filebeat-saving-what-files-filebeat-has-processed-in-es/276097>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 16, 2021, 8:03am UTC](https://discuss.elastic.co/t/filebeat-saving-what-files-filebeat-has-processed-in-es/276097 "2021-06-16T08:03:24Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![scott\_stash](https://avatars.discourse-cdn.com/v4/letter/s/c89c15/32.png) [@scott\_stash](https://discuss.elastic.co/u/scott_stash)\
**Post date:** [June 16, 2021, 8:03am UTC](https://discuss.elastic.co/t/filebeat-saving-what-files-filebeat-has-processed-in-es/276097/1 "2021-06-16T08:03:25Z")

</div>

I have a system setup where log files in a folder are processed by FB-\> LS -ES, once they are processed some reports/scripts are ran for the data in ES and the output is sent to a user.

To accomplish this I have written a python service that checks the files/filesize on the OS, and compares this to what is in the Filebeat Registry. When the Filebeat Registry offset ='s the OS size in bytes file is marked as complete. The service creates a document in ES for each file it finds/scans with information such as processed true/false, and current fb/offset.

Does anyone do anything similar? And if so how do you accomplish this.

I'm wondering if it would just be easier to add this to Filebeat:  
-when File is opened create an entry in ES for the filename, processing\_complete ='s false.  
-file finishes harvesting have filebeat post to ES saying file is complete

Perhaps there is already a way to do this though? It would seem like having a backup of the registry in ES, or something similar would be not that un-ordinary of an ask. (Maybe its buried in some monitoring data?)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 14, 2021, 10:04am UTC](https://discuss.elastic.co/t/filebeat-saving-what-files-filebeat-has-processed-in-es/276097/2 "2021-07-14T10:04:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.