S3 input plugin - pull the files from only specific folder

Elastic Stack Logstash
1

I want to index the files from S3 to elasticsearch. I am not able to see the option of specifying the folder in the bucket for s3 input plugin. Is there any option to mention only the specific path instead of pulling the files from the complete bucket?

2

Is it possible to mention the specific path along with the S3 bucket,as we don't want to pull all the data from all folders from the bucket.

3

You can use the prefix property for this:

input {
  s3 {
    # ...
    prefix => "/path/to/files/you/want/"
  }
}
4

Thanks. Tried the prefix option and none of the data from the files ( within folders ) are getting pulled. But when I comment out this statement and only the bucket property is present then I could see it is working. So still checking how to make it work for a specific folder.

5

Looks like the prefix parameter is for the files within the buckets and not for the folder.

If specified, the prefix of filenames in the bucket must match (not a regexp)

I want something like this.
/bucket name/datewise folder name/all files needs to be read

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.