# Adding S3 Bucket Name in Output Logs

**URL:** <https://discuss.elastic.co/t/adding-s3-bucket-name-in-output-logs/110385>\
**Category:** Logstash\
**Created:** [December 5, 2017, 3:24pm UTC](https://discuss.elastic.co/t/adding-s3-bucket-name-in-output-logs/110385 "2017-12-05T15:24:41Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![jmukhtar](https://avatars.discourse-cdn.com/v4/letter/j/a9a28c/32.png) [@jmukhtar](https://discuss.elastic.co/u/jmukhtar)\
**Post date:** [December 5, 2017, 3:24pm UTC](https://discuss.elastic.co/t/adding-s3-bucket-name-in-output-logs/110385/1 "2017-12-05T15:24:41Z")

</div>

I am a newbie to Logstash and ES in general, and struggling to add S3 bucket name to logstash output.

I have a setup where we have multiple s3 buckets with RDS instance logs (1 bucket per instance) and we want to harvest the logs from the bucket.

In order to identify the instance, we need to add bucket name to the logstash output but i can't find a way to do that. I am able to add file name via [@metadata][s3][key] but what i need is the bucket name.

Can anyone help me in doing that?

Thanks,  
Junaid

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 8, 2017, 6:17am UTC](https://discuss.elastic.co/t/adding-s3-bucket-name-in-output-logs/110385/2 "2017-12-08T06:17:24Z")

</div>

Aren't you setting the bucket name in the s3 input confguration (please always post your configuration when you ask a question)? Then just use `add_field` in the input to add the same string as a field.

---

<div class="post-metadata">

**Author:** ![jmukhtar](https://avatars.discourse-cdn.com/v4/letter/j/a9a28c/32.png) [@jmukhtar](https://discuss.elastic.co/u/jmukhtar)\
**Post date:** [December 11, 2017, 2:59pm UTC](https://discuss.elastic.co/t/adding-s3-bucket-name-in-output-logs/110385/3 "2017-12-11T14:59:55Z")

</div>

Details are listed below

S3 Bucket and Structure

Bucket Name: production-logs  
Folder Name: Instance\_A/{{Log\_Types}/Log\_Files ==\> Log\_Types is error/debug/other  
Folder Name: Instance\_B/{{Log\_Types}/Log\_Files ==\> Log\_Types is error/debug/other  
Folder Name: Instance\_C/{{Log\_Types}/Log\_Files ==\> Log\_Types is error/debug/other  
Folder Name: Instance\_D/{{Log\_Types}/Log\_Files ==\> Log\_Types is error/debug/other

Bucket Name: preproduction-logs  
Folder Name: Instance\_A/{{Log\_Types}/Log\_Files ==\> Log\_Types is error/debug/other  
Folder Name: Instance\_B/{{Log\_Types}/Log\_Files ==\> Log\_Types is error/debug/other  
Folder Name: Instance\_C/{{Log\_Types}/Log\_Files ==\> Log\_Types is error/debug/other  
Folder Name: Instance\_D/{{Log\_Types}/Log\_Files ==\> Log\_Types is error/debug/other

below is what i am currently doing to set the name  
input {  
s3 {  
access\_key\_id =\> "access\_key\_id"  
bucket =\> "production-logs"  
region =\> "eu-west-1"  
secret\_access\_key =\> "secret\_access\_key"  
type =\> "s3"  
sincedb\_path =\> "/dev/null"  
prefix =\> "Instance\_A/error/"  
add\_field =\> { "[@metadata][buck\_name]" =\> "Instance\_A-error-logs" }  
}  
}  
filter {  
grok {  
match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:timestamp}%{SPACE}%{TZ}::@:[%{DATA:PID}]%{SPACE}:%{WORD:lvl}:%{GREEDYDATA} " }  
}  
mutate {  
lowercase =\> ["lvl"]  
remove\_field =\> ["message"]  
add\_field =\> {  
"hostname" =\> "%{[@metadata][buck\_name]}"  
}  
}  
}

This is a time consuming and error prone method and doesn't give me flexibility to add more instances to it dynamically.

If i have the option of adding "S3 bucket name" or "Folder name" dynamically then i can skip better handle the inputs

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 11, 2017, 3:07pm UTC](https://discuss.elastic.co/t/adding-s3-bucket-name-in-output-logs/110385/4 "2017-12-11T15:07:27Z")

</div>

> This is a time consuming and error prone method and doesn't give me flexibility to add more instances to it dynamically.

You'd obviously want to generate the configuration files.

> If i have the option of adding "S3 bucket name" or "Folder name" dynamically then i can skip better handle the inputs

Yes, but that's not possible right now.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 8, 2018, 3:07pm UTC](https://discuss.elastic.co/t/adding-s3-bucket-name-in-output-logs/110385/5 "2018-01-08T15:07:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.