Hi,
I'm using the S3 input plug-in to read log files from a S3 folder.
This is generally working well but from time to time Logstash skips files. I'm using sincedb to keep track of the processes files and move and delete the original file after Logstash is done processing them. I've got the interval set to 1 minute.
I'm not seeing any errors in the logs and the files that are skipped processes just fine if I delete & re-upload them so I'm expecting some kind of timing issue where maybe a log gets uploaded to S3 while Logstash is still processing another log and then somehow doesn't pick up the new file.
Is there any way to confirm whether this is indeed a timing issue? and if it is, how could I solve this?
Is not using sincedb an option? As long as the move & delete option works I don't see that being much of an issue. Of course if Logstash would stop unexpectedly some logs might be ingested twice but Logstash has been stable enough for me not to worry too much about that.
Not using sincedb doesn't mean Logstash will start reading the same Log file again while it is still being processed, correct?