Hi - I'm using the logstash S3 input plugin - and need to implement "backup_add_prefix" - so that I can skip processed files - to improve performance.
I assumed the first step to implementing this, was to get the prefix option working.
All of the files in our S3 bucket have the same numeric prefix - however when I set this as the prefix
I get:
No files found in bucket {:prefix=>"123456789"}
I'm a bit confused how this prefix works - can some one tell me how it is intended to work - the docs say it is a filename prefix - but I have seen examples that look like people are using it as a path.
If it is a filename prefix, am I setting it wrong?
I would expect a prefix of 123456789 to match files that start with this
i.e.
123456789_somefile.log
123456789_someotherfile.log
--Update--
I've configured the prefix as a path (tried this previously, but found I had to remove the leading /) - and logstash is finding the files. However, this means that my use-case wont work.
I have found multiple discussions saying that the prefix is used to select the files, and the backup_add_prefix renames the files, so they wont be picked up by the prefix next interval.
Is this not the case?
Thanks
Dave