S3 input plugin problem with %

Elastic Stack Logstash
1

Hi, we use S3 input plugin and we have a problem with backup of indexed files. Our files contain character % because URL encoded format. Indexing is OK but than when file should be moved to the archive we receive errors:
Error: The specified key does not exist. and Exception: Aws::S3::Errors::NoSuchKey
The plugin is restarted file indexed again and error. It stops the whole pipeline, and new files are not indexed ...

Is there any way to use this plugin with backup option and character % in the name of the file?

Thank you

2

Hello,

Please share your pipeline configuration and the exact log you are getting.

3

image
image772×1024 66 KB

4

[2025-01-07T13:43:23,555][ERROR][logstash.javapipeline ][s3-cognito][ae84138f2292f0daed2086b40adbc1c94794db398deff4feb67588bc25e239f5] A plugin had an unrecoverable error. Will restart this plugin.

Error: The specified key does not exist. WSO2-LOGSTASH
January 07, 2025 at 14:43 (UTC+1:00) Exception: Aws::S3::Errors::NoSuchKey
5

You would have more log lines around the same time, please share them as well.

If I understood correctly you can download objects from the s3 bucket and logstash can process them.

But since you have the backup_to_bucket option, it cannot write to the bucket because your filenames have special characters that may be causing some issue?

Logstash uses the AWS SDK, not sure if you can solve this in Logstash as you would need to rename the files with the special characters and there is no option to do that.