Logstash s3 output plugin folder structure

Elastic Stack Logstash
1

Our DevOps engineers have been using Logstash S3 plugin which simply puts all data in a S3 bucket location. Since we have configured files to be created in every hour on S3, the number of files in the S3 location touched thousand in just one and a half month. We decided to store the files in YYYY/MM/DD folder structure. For example, files created on date 2016-06-16 would go inside s3://location/2016/06/16. These YYYY, MM, and DD folder would get created dynamically.

i found few discussions over the forums speaking about tweaking the s3.rb file by adding:

t = Time.new
date_s3 = t.strftime(“%Y/%m/%d/”)

and replacing the remote_filename value to

remote_filename = “#{@prefix}#{date_s3}#{File.basename(file)}”

but i guess this option was deprecated through the newer versions of s3 plugin, since i cannot find remote_filename in s3.rb

There is any other solution for this request?

2

Try

prefix => "%{+YYYY}/%{+MM}/%{+dd}"

That will create a S3://myBucket/2021/01/14/ structure dynamically.

3

You don't get a lot of options for custom filenames in s3 output plugin.

There is an option to tag the data in the s3 output plugin that will add the tag to the file name.

See

Add the following to your s3 output plugin parameters.

tags => ["some_tag"]
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.