Hello,
My logstash runs on EC2 and harvest data from s3. For some reasons, the logstash host machine is sometimes being terminated and replaced with new one (same AMI). In such cases lohstash's sincedb file is lost and when new machine is created logstash harvest data from the entire bucket and not just new data. I know there is a possibility to locate the since db file on some persistent drive (like EBS), but for saving cost I tried to use my existing s3 bucket. What I tried to do is to copy the sincedb file to s3 when the instance is terminated, and then copy it back to custom sincedb_path when new machine is initiated. It worked quite well, I see no errors, logstash can even modify the file I provide, but it still reads the entire s3 bucket... What can be wrong? Is it even possible to do what I try to do?