I'm using the Logstash s3 plugin to ingest logs from a partner's S3 bucket.
Logstash itself runs in a container, so its sincedb is not persistent. When the container re-starts, Logstash restarts and begins ingesting logs from that bucket since the beginning of time.
backup_to_bucket is not really an option because it's not my bucket to manage.
Is there a way I can tell Logstash to ignore anything older than ~1 day?
Can I seed sincedb with a date? e.g. before Logstash starts, can I echo a date in the format
2020-01-31 07:01:33 +0000 (where that date is ~one day ago) to:
What is the syntax for the hash on the name of the sincedb file? e.g.
sincedb_ca090558edfcc5759ac626c813a5a2c2. Or I can just make this whatever I want and use
Any other suggestions? Thanks in advance. -Clark