I have data that are coming daily. 90% of the data are same as previous day. 5% are new and 5% are gone. there is no way I can find out what is gone and what is new.
this are list of dirs from one of our busy system where we have million directory, just need to save their name and full path. some gets remove some gets added and rest stays as is.
for example.
/home/user1
/home/user1/data
/home/user1/mylogic
/home/user2/data
/home/user2/whatever
I get dump in text file for all of this dirs.
my thinking was to just delete old index and recreate same name index daily.
can I do this via logstash?
Is there better way to achieve this?