Input S3 does NOT work properly with prefix option

Elastic Stack Logstash
1

Hello,

I am trying to set up a logstash pipeline with input source AWS S3. I set up as below and tested, and saw all log files were successfully retrieved by logstash. but only problem was from using "prefix" option.

input {
  s3 {
    bucket => "xxx"
    region => "xxx"
    access_key_id => "xxx"
    secret_access_key => "xxx"
    id => "xxx"
    **prefix =>** "sample-log/E1GL1WCR8XPQ2H*"
    type => "s3"
  }
#  stdin {}
}

the source S3 structure looks attached image (I captured from AWS console).

1
1.JPG1320×527 61.8 KB

I put prefix option as above but got INFO line as below. It seemed logstash couldn't find file with prefix.

[2019-08-23T01:30:25,410][INFO ][logstash.inputs.s3 ] S3 input: No files found in bucket {:prefix=>"sample-log/E1GL1WCR8XPQ2H*"}

was my above input code something wrong?

the log file from S3 should be indexed into ES based on the prefix of the log file name.

please advise.

thank you!

[I am using Logstash 6.8.2]

1 Like
2

The prefix should not be a regexp. I believe the trailing * is implied.

3

thank you for your reply. :slight_smile:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.