I have a system setup where log files in a folder are processed by FB-> LS -ES, once they are processed some reports/scripts are ran for the data in ES and the output is sent to a user.
To accomplish this I have written a python service that checks the files/filesize on the OS, and compares this to what is in the Filebeat Registry. When the Filebeat Registry offset ='s the OS size in bytes file is marked as complete. The service creates a document in ES for each file it finds/scans with information such as processed true/false, and current fb/offset.
Does anyone do anything similar? And if so how do you accomplish this.
I'm wondering if it would just be easier to add this to Filebeat:
-when File is opened create an entry in ES for the filename, processing_complete ='s false.
-file finishes harvesting have filebeat post to ES saying file is complete
Perhaps there is already a way to do this though? It would seem like having a backup of the registry in ES, or something similar would be not that un-ordinary of an ask. (Maybe its buried in some monitoring data?)