Ignore all files in s3 and read only current files from S3

Elastic Stack Logstash
1

how to ignore old files and push only latest log files from S3 using logstash. We are using logstash to push cloudtaril logs from s3 to elasticsearch. Cloudtrail logs are in below format

/AWSLogs/CloudTrail/xxxAccount Numberxxxx/aws-region/year(YYYY)/Month(MM)/day(DD)/

I need to pull only latest data(like data form current month), as the entire bucket has huge terrabytes of data and logstash is not able to scale that much data. Is there a way to do this?

2

have a try with start_position and sincedb_path

3

@larrylui I tried them, but logstash is listing all the files everytime first and then push data from the timestamp set in sincedb path. But even listing files is taking 3-4hrs everytime.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.