# Dynamic Bucket Names or Directories in AWS S3 Output **URL:** \ **Category:** Logstash\ **Created:** [June 11, 2015, 12:42pm UTC](https://discuss.elastic.co/t/dynamic-bucket-names-or-directories-in-aws-s3-output/2459 "2015-06-11T12:42:58Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![umutcan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/umutcan/32/625_2.png) [@umutcan](https://discuss.elastic.co/u/umutcan)\ **Post date:** [June 11, 2015, 12:42pm UTC](https://discuss.elastic.co/t/dynamic-bucket-names-or-directories-in-aws-s3-output/2459/1 "2015-06-11T12:42:58Z")
Hi, I am doing some tests on storing data on AWS S3. I have read the documents and couldn't find a dynamic bucket name or directory option like Elasticsearch output provides in index name. Is there a way to do this? Thank you ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 13, 2015, 5:01am UTC](https://discuss.elastic.co/t/dynamic-bucket-names-or-directories-in-aws-s3-output/2459/2 "2015-06-13T05:01:36Z")
You can use exactly the same method for that output. ---
**Author:** ![umutcan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/umutcan/32/625_2.png) [@umutcan](https://discuss.elastic.co/u/umutcan)\ **Post date:** [June 17, 2015, 12:00pm UTC](https://discuss.elastic.co/t/dynamic-bucket-names-or-directories-in-aws-s3-output/2459/3 "2015-06-17T12:00:53Z")
Can you provide an example for it? I tried it several ways and it doesn't work. I am using 1.5.0 version of Logstash. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 18, 2015, 2:04am UTC](https://discuss.elastic.co/t/dynamic-bucket-names-or-directories-in-aws-s3-output/2459/4 "2015-06-18T02:04:49Z")
Providing what you have tried would be useful 🙂 ---
**Author:** ![tristan](https://avatars.discourse-cdn.com/v4/letter/t/96bed5/32.png) [@tristan](https://discuss.elastic.co/u/tristan)\ **Post date:** [June 23, 2015, 9:54pm UTC](https://discuss.elastic.co/t/dynamic-bucket-names-or-directories-in-aws-s3-output/2459/5 "2015-06-23T21:54:03Z")
I've been trying to get this to work also. My dev set up works just fine with a config file of ``` input { file { path => "/srv/log/app/server/*.log" } } filter { grok { match => ["path","%{GREEDYDATA:folder}/%{GREEDYDATA:filename}\.log"] } } output { s3 { bucket => "test" prefix => "test/" size_file => 2048 time_file => 5 canned_acl => "private" codec => rubydebug } stdout { codec => rubydebug } } ``` I'd like to be able to use a wild card in the prefix to do something like prefix =\> "test/%{folder}", which the docs made me think might work, but that doesn't seem to work. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 24, 2015, 5:49am UTC](https://discuss.elastic.co/t/dynamic-bucket-names-or-directories-in-aws-s3-output/2459/6 "2015-06-24T05:49:16Z")
> I'd like to be able to use a wild card in the prefix to do something like prefix =\> "test/%{folder}", which the docs made me think might work, but that doesn't seem to work. Sorry, that won't work since `%{varname}` interpolation doesn't take place for the `prefix` parameter's value (the highlighted line indicates that we're using the raw parameter value, `@prefix`, instead of `event.sprintf(@prefix)`): > However, fixing this is probably not entirely simple since it would mean that the output file could potentially change between every single message received by the output. ---
**Author:** ![tristan](https://avatars.discourse-cdn.com/v4/letter/t/96bed5/32.png) [@tristan](https://discuss.elastic.co/u/tristan)\ **Post date:** [June 25, 2015, 12:50am UTC](https://discuss.elastic.co/t/dynamic-bucket-names-or-directories-in-aws-s3-output/2459/7 "2015-06-25T00:50:45Z")
Thanks, Now that I think about it I see how that would be a huge problem. ---
**Author:** ![haroldwoo](https://avatars.discourse-cdn.com/v4/letter/h/a9adbd/32.png) [@haroldwoo](https://discuss.elastic.co/u/haroldwoo)\ **Post date:** [August 12, 2015, 7:40pm UTC](https://discuss.elastic.co/t/dynamic-bucket-names-or-directories-in-aws-s3-output/2459/8 "2015-08-12T19:40:45Z")
How about the other way around? Is logstash able to support dynamic bucket names in s3 input similar to how it does for file inputs? e.g. input { s3 { bucket =\> "logbucket" prefix =\> "logs/\*/2015/01/01/" } } I want to use a grok filter on the s3 prefix to add fields to my log entries grok { match =\> ["prefix", "logs/%{GREEDYDATA:projectName/2015/01/01/"] } ---
**Author:** ![matt.koivisto](https://avatars.discourse-cdn.com/v4/letter/m/7ba0ec/32.png) [@matt.koivisto](https://discuss.elastic.co/u/matt.koivisto)\ **Post date:** [February 8, 2016, 1:46pm UTC](https://discuss.elastic.co/t/dynamic-bucket-names-or-directories-in-aws-s3-output/2459/9 "2016-02-08T13:46:53Z")
Since my set of prefix's was known, I worked around this by putting if conditions and hard coded prefixes: ``` output { # Until s3 output supports variables in prefix if [fields][host] == "foohost" { s3 { access_key_id => "" secret_access_key => "" bucket => "host-logs" time_file => 60 prefix => "foohost" } } if [fields][host] == "barhost" { s3 { access_key_id => "" secret_access_key => "" bucket => "host-logs" time_file => 60 prefix => "barhost" } } ... } ``` ---
**Author:** ![xxDECKERxx](https://avatars.discourse-cdn.com/v4/letter/x/ed8c4c/32.png) [@xxDECKERxx](https://discuss.elastic.co/u/xxDECKERxx)\ **Post date:** [February 15, 2017, 8:57pm UTC](https://discuss.elastic.co/t/dynamic-bucket-names-or-directories-in-aws-s3-output/2459/10 "2017-02-15T20:57:03Z")
I am currently trying to do something similar with my s3 output. I have input configurations coming from multiple file locations, and depending on the directory the logs come from I am set the "type" field to a specific value. Is there no way to use a field value as a prefix or part of the tags? I need to make different s3 bucket objects based on the type otherwise they all get written to the same object in s3. The work-around would be to use if conditions, but it would be simpler to be able to use references in the s3 output. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:28am UTC](https://discuss.elastic.co/t/dynamic-bucket-names-or-directories-in-aws-s3-output/2459/11 "2017-07-06T04:28:34Z")