Hi
I have rotating logs i.e. 11 log files each of some fixed size. They are filled from file 1 then 2 then 3 and till 11 after that the new logs overwrites the old logs in file 1. And we are planning to run the logstash configuration file at some regular intervals say for every 1 day.
But in that time span 90% of our machines have only 5-10 % of new log records of all the records in 11 log files.
So, there is no point in feeding all the logs or comparing them with the existing records to determine whether it will be inserted or not causes huge overhead
If only half of the first file is been updated then is there any way that when I read those next time it's starts reading from position where it has been last time and after completion of that file it should read next file.
Thank u