# S3 dynamic folder naming in yyyy/mm/dd

**URL:** <https://discuss.elastic.co/t/s3-dynamic-folder-naming-in-yyyy-mm-dd/58879>\
**Category:** Logstash\
**Created:** [August 25, 2016, 2:19am UTC](https://discuss.elastic.co/t/s3-dynamic-folder-naming-in-yyyy-mm-dd/58879 "2016-08-25T02:19:27Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![lrem006](https://avatars.discourse-cdn.com/v4/letter/l/7ba0ec/32.png) [@lrem006](https://discuss.elastic.co/u/lrem006)\
**Post date:** [August 25, 2016, 2:19am UTC](https://discuss.elastic.co/t/s3-dynamic-folder-naming-in-yyyy-mm-dd/58879/1 "2016-08-25T02:19:27Z")

</div>

Someone on another website proposed a mechanism for dynamically creating folders in the YYYY/mm/dd for the s3 output plugin. Effectively changing the function "write\_on\_bucket" to the following starting few lines. This was suggested from the following website:

> **[Tweaking Logstash’s S3 plugin to create folders in YYYY/MM/DD format on AWS S3 |...](http://www.tothenew.com/blog/tweaking-logstashs-s3-plugin-to-create-folders-in-yyyymmdd-format-on-aws-s3/)**
>
> Logstash is a service that accepts logs from a variety of systems, processes it and allows us to index it in Elasticsearch etc which can be visualised using Kibana. Our DevOps engineers have been using Logstash S3 plugin which simply puts all data in...

def write\_on\_bucket(file)  
# find and use the bucket  
bucket = @s3.buckets[@bucket]  
t = Time.new  
date\_s3 = t.strftime("%Y/%m/%d/")  
remote\_filename = "#{@prefix}#{date\_s3}#{File.basename(file)}"

This worked! It created the subfolders provided a prefix was provided as suggested on the website. So what is the problem? Well two problems actually:

1. The logstash-programmatic-access-test-object\* will not delete itself
2. when specifying size\_file in the output plugin, it will increment up with each iteration; even if singularly the file size is small

Does anyone know what might be going on?

---

<div class="post-metadata">

**Author:** ![lrem006](https://avatars.discourse-cdn.com/v4/letter/l/7ba0ec/32.png) [@lrem006](https://discuss.elastic.co/u/lrem006)\
**Post date:** [August 25, 2016, 2:19pm UTC](https://discuss.elastic.co/t/s3-dynamic-folder-naming-in-yyyy-mm-dd/58879/2 "2016-08-25T14:19:21Z")

</div>

Regarding #1, it was necessary to change remote\_filename on the delete function as well, so this is no longer an issue. Still tracking the size file incrementing

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:41am UTC](https://discuss.elastic.co/t/s3-dynamic-folder-naming-in-yyyy-mm-dd/58879/3 "2017-07-06T04:41:37Z")

</div>