Logstash s3 input plugin not deleting files after processing

Elastic Stack Logstash
1

I am using s3 input plugin on logstash to read AWS VPC flow logs from s3 bucket. I have this as my input.conf file

input {
s3 {
bucket => "vpc-flow-logs-elk"
prefix => "AWSLogs/200312345677/vpcflowlogs/us-east-1/"
region => "us-east-1"
delete => true
include_object_properties => true
interval => 10
type => "vpcflowlogs"
}
}

The files are being indexed. But not deleted after processing. I do have the s3:DeleteObject iam permissions and have tried manually deleting few files from the EC2 instance

aws s3 rm s3://file_name works

Not sure why logstash is not deleting the objects. How can i verify that? Can you please advise accordingly.

Thank you.

3

The above code works. The s3 bucket had so many objects and hence it took a really long time to index them before deleting.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.