# Create directories according to field in s3 bucket

**URL:** <https://discuss.elastic.co/t/create-directories-according-to-field-in-s3-bucket/111173>\
**Category:** Logstash\
**Created:** [December 11, 2017, 8:36pm UTC](https://discuss.elastic.co/t/create-directories-according-to-field-in-s3-bucket/111173 "2017-12-11T20:36:35Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ferdous\_Shibly](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ferdous_shibly/32/23470_2.png) [@Ferdous\_Shibly](https://discuss.elastic.co/u/Ferdous_Shibly)\
**Post date:** [December 11, 2017, 8:36pm UTC](https://discuss.elastic.co/t/create-directories-according-to-field-in-s3-bucket/111173/1 "2017-12-11T20:36:35Z")

</div>

Hi,

We are using logstash to route logs to different places like different elasticsearch clusters. Now we want to store logs in s3 bucket as well. We are using logstash-output-s3. logstash-output-s3 plugin could store all the logs in s3 bucket. but we want to create different buckets based on fields. Here is our logstash configuration

```
output {

  s3{

    region => "us-east-1"

    bucket => "mycompany-qa-application-logs"

    prefix => "app/%{[attribute][APPLICATION]}-%{[attribute][ENVIRONMENT]}-%{[logType]}-logs"

    size_file => 2146304

    time_file => 5

    canned_acl => "private"

  }

} 

```

it should create app-qa-error-logs. but it created %{[attribute][APPLICATION]}-%{[attribute][ENVIRONMENT]}-%{[logType]}-logs under app directory.

Any help will be much appreciated.

Cheers  
Ferdous Shibly

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 12, 2017, 9:57am UTC](https://discuss.elastic.co/t/create-directories-according-to-field-in-s3-bucket/111173/2 "2017-12-12T09:57:15Z")

</div>

How is the `attribute` field supposed to be populated?

---

<div class="post-metadata">

**Author:** ![Ferdous\_Shibly](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ferdous_shibly/32/23470_2.png) [@Ferdous\_Shibly](https://discuss.elastic.co/u/Ferdous_Shibly)\
**Post date:** [December 12, 2017, 4:25pm UTC](https://discuss.elastic.co/t/create-directories-according-to-field-in-s3-bucket/111173/3 "2017-12-12T16:25:19Z")

</div>

attribute field is already in the logs. We use json filter to separate out. We used

> prefix =\> "app/%{[attribute][APPLICATION]}-%{[attribute][ENVIRONMENT]}-%{[logType]}-logs"

which is creating the directories accordingly. Now we are facing a new challenge. Some of our applications doesn't have logType field in their logs. How can we send the logs to error directory if the logType is missing, for example, app1 doesn't have logType, so the logs will be sent to app1-error-logs directory?

Thanks for your help.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 12, 2017, 7:31pm UTC](https://discuss.elastic.co/t/create-directories-according-to-field-in-s3-bucket/111173/4 "2017-12-12T19:31:15Z")

</div>

> <https://stackoverflow.com/questions/30309096/logstash-check-if-field-exists>

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 9, 2018, 7:31pm UTC](https://discuss.elastic.co/t/create-directories-according-to-field-in-s3-bucket/111173/5 "2018-01-09T19:31:15Z")

</div>

Closed on January 9, 2018, 7:31pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.