Exclude_pattern in logstash s3 input

Elastic Stack Logstash
1

This is my s3 input config. I am trying to exclude a key that has name called CloudTrail-Digest.

input {
  s3 {
    type => "cloudtrail"
    bucket => "aws"
    prefix => "Accounts/"
    add_field => { source => gzfiles }
    backup_to_dir => "/mnt/aws/s3backup"
    exclude_pattern => "^CloudTrail-Digest/"
    codec => cloudtrail {}
    region => "us-west-2"
    sincedb_path => "/etc/s3backup/sincedb"
  }
}

But when i try to run logstash. It still downloads that file. Any idea if my regex is wrong.

20:47:01.414 [[main]<s3] DEBUG logstash.inputs.s3 - S3 input: Adding to objects[] {:key=>"Accounts/bigdata/AWSLogs/xxxxxxxxxxx/CloudTrail-Digest/ap-northeast-2/2017/07/12/xxxxxxxxxxxx_CloudTrail-Digest_ap-northeast-2_CybersecurityTemplate-CloudTrailImpl-11EE2N5EAOL3B_us-west-2_20170712T090720Z.json.gz"}
20:47:01.414 [[main]<s3] DEBUG logstash.inputs.s3 - objects[] length is:  {:length=>2999}
2

I'm not familiar with the s3 input but: ^CloudTrail-Digest/ matches strings that begin with CloudTrail-Digest, which the path in question clearly doesn't.

3

So how can i exclude all the files that start with CloudTrail-Digest?

4

You can use /CloudTrail-Digest/ to exclude all files that has CloudTrail-Digest as one patch component.

5

I will try this one and update you.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.