I'm using the Logstash s3 plugin to ingest logs from a partner's S3 bucket. Logstash itself runs in a container, so its sincedb is not persistent. When the container re-starts, Logstash restarts and begins ingesting logs from that bucket since the beginning of time. delete or backup_to_bucket is …

Tell Logstash to ignore old files in s3 bucket

Badger April 6, 2020, 7:04pm 2

This thread is about dropping events using the output of an age{} filter. It would still read the bucket, but not reprocess all the data.

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.