I have the following config ,
- /var/lib/transfers/report//archive/processed/ABC-Reconciliation.csv --> which is new reports files everyday
Each day new files with different name file for one of the defined prospector is generated (15 files) . When i ran it for first time , it picked all the 90 days file and got some error too many open files for harvesting and at moment redis also stopped working . I cleared the registry file and kept it as empty and re-ran, still i saw it picked some random old dated files .
It doesnt pick the files which is not older than 24 hours .
Now none of the files are picked and it stopped completely , only it works for sometime whenever we restart the
filebeat and after sometime it stops harvesting the files .
P.S. : - /home/a/b/logs/check.log --> this is dynamic logfile and rotated everyday
How can i make it work as i configured ?