# Logstash s3 output plugin folder structure

**URL:** <https://discuss.elastic.co/t/logstash-s3-output-plugin-folder-structure/261152>\
**Category:** Logstash\
**Tags:** docker\
**Created:** [January 14, 2021, 4:21pm UTC](https://discuss.elastic.co/t/logstash-s3-output-plugin-folder-structure/261152 "2021-01-14T16:21:47Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ohad\_Elias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ohad_elias/32/82256_2.png) [@Ohad\_Elias](https://discuss.elastic.co/u/Ohad_Elias)\
**Post date:** [January 14, 2021, 4:21pm UTC](https://discuss.elastic.co/t/logstash-s3-output-plugin-folder-structure/261152/1 "2021-01-14T16:21:47Z")

</div>

Our DevOps engineers have been using Logstash S3 plugin which simply puts all data in a S3 bucket location. Since we have configured files to be created in every hour on S3, the number of files in the S3 location touched thousand in just one and a half month. We decided to store the files in YYYY/MM/DD folder structure. For example, files created on date 2016-06-16 would go inside s3://location/2016/06/16. These YYYY, MM, and DD folder would get created dynamically.

i found few discussions over the forums speaking about tweaking the s3.rb file by adding:

```auto
t = Time.new
date_s3 = t.strftime(“%Y/%m/%d/”)

```

and replacing the remote\_filename value to

```auto
remote_filename = “#{@prefix}#{date_s3}#{File.basename(file)}”

```

but i guess this option was deprecated through the newer versions of s3 plugin, since i cannot find remote\_filename in s3.rb

There is any other solution for this request?

---

<div class="post-metadata">

**Author:** ![okgnae](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/okgnae/32/82212_2.png) [@okgnae](https://discuss.elastic.co/u/okgnae)\
**Post date:** [January 14, 2021, 4:35pm UTC](https://discuss.elastic.co/t/logstash-s3-output-plugin-folder-structure/261152/2 "2021-01-14T16:35:19Z")

</div>

Try

```auto
prefix => "%{+YYYY}/%{+MM}/%{+dd}"

```

That will create a `S3://myBucket/2021/01/14/` structure dynamically.

---

<div class="post-metadata">

**Author:** ![okgnae](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/okgnae/32/82212_2.png) [@okgnae](https://discuss.elastic.co/u/okgnae)\
**Post date:** [January 14, 2021, 4:43pm UTC](https://discuss.elastic.co/t/logstash-s3-output-plugin-folder-structure/261152/3 "2021-01-14T16:43:47Z")

</div>

You don't get a lot of options for custom filenames in s3 output plugin.

There is an option to tag the data in the s3 output plugin that will add the tag to the file name.

See

> **[S3 output plugin | Logstash Reference \[7.10\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-s3.html#_s3_output_file)**

Add the following to your s3 output plugin parameters.

```auto
tags => ["some_tag"]

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 11, 2021, 4:43pm UTC](https://discuss.elastic.co/t/logstash-s3-output-plugin-folder-structure/261152/4 "2021-02-11T16:43:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.