Logstash S3 input plugin - prefix usage

Hi - I'm using the logstash S3 input plugin - and need to implement "backup_add_prefix" - so that I can skip processed files - to improve performance.

I assumed the first step to implementing this, was to get the prefix option working.

All of the files in our S3 bucket have the same numeric prefix - however when I set this as the prefix

I get:
No files found in bucket {:prefix=>"123456789"}

I'm a bit confused how this prefix works - can some one tell me how it is intended to work - the docs say it is a filename prefix - but I have seen examples that look like people are using it as a path.

If it is a filename prefix, am I setting it wrong?

I would expect a prefix of 123456789 to match files that start with this

I've configured the prefix as a path (tried this previously, but found I had to remove the leading /) - and logstash is finding the files. However, this means that my use-case wont work.

I have found multiple discussions saying that the prefix is used to select the files, and the backup_add_prefix renames the files, so they wont be picked up by the prefix next interval.

Is this not the case?


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.