Parsing S3 bucket into Logstash

Elastic Stack Logstash
1

Hi Team,

when I am trying to use the S3 input plugin.

S3 location:s3://bucket123/DAG/TASK/2020-04-26T16:01:00+00:00/1.log
'''
The resulting error is :
S3 input: Unable to download remote file {:remote_key=>"bucket123/DAG/TASK/2020-04-26T16:01:00 00:00/1.log", :message=>"The specified key does not exist."}
'''
'''
This issue only occurs when the key consists of the "+" symbol which seems to be replaced with a space when returned to logstash.'''

The key can be found but the file just cannot be downloaded.

Can you please help and provide guidance on this?

2

Guys , any link on this?
is this possible in logstash?

3

Hi @SunilYadav - Welcome to our community forums!

As mentioned in Object key and metadata, the + character in a key name might require additional code handling and likely need to be URL encoded or referenced as HEX.

Could you try to change the prefix value accordingly? You could use this online tool to get the encoded string. For example:

  • XXXX/yyyy/2020-06-27T12:00:00+00:00 would be become XXXX%2Fyyyy%2F2020-06-27T12%3A00%3A00%2B00%3A00

Let us know if that works.

Note: FYI, there is an existing issue for the same problem in the respective Github repository.

4

Hi Romain,
Thanks for your reply!

We have tried this already and its not working giving the error "no file found in bucket".

What are the other options available?

Thanks,
Sunil Yadav

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.