I'm using the Logstash s3 plugin to ingest logs from a partner's S3 bucket. Logstash itself runs in a container, so its sincedb is not persistent. When the container re-starts, Logstash restarts and begins ingesting logs from that bucket since the beginning of time. delete or backup_to_bucket is …

Tell Logstash to ignore old files in s3 bucket

Badger April 6, 2020, 7:04pm 2

This thread is about dropping events using the output of an age{} filter. It would still read the bucket, but not reprocess all the data.

Topic		Replies	Views
S3 input plugin. Parse again an S3 bucket Logstash	3	628	September 11, 2019
Provided sincedb file is ignored Logstash	2	23	September 20, 2024
Logstash s3 input reads file multiple times Logstash	1	198	July 21, 2023
Logstash sometimes skipping files on S3 Logstash	1	566	October 29, 2018
Logstash S3 Input - How does it know where to start? Logstash	2	1562	May 8, 2018

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.

Tell Logstash to ignore old files in s3 bucket

Related topics