Hello,
So we have a scenario where every day some files have been placed in a directory in a server. Now we just need to parse the files , extract relevant information from logs and send it to elasticsearch.
The problem is that once the whole thing is done, we need to be able to archive/delete the files in the directory. Now we can have another script that can just delete the files but the question is that how do we know that logstash has now finished processing all files.
Is there any EOF (end of file finished ) processing marker left by logstash somewhere. or we know that now the file has been read properly and we can trigger our some custom script from logstash to archive the files.
We can use filebeat as it maintains a counter in a back end file to indicate the amount of log it has read. and in our parallel batch script e.t.c we can continuously read that file to see if all files have finished processing and once the filebeat has stopped updating that file , it means it has been done and we can just go ahead and delete the files or move them to archive directory,
What solution would you guys recommend in order to achieve this.