Adding S3 Bucket Name in Output Logs

[jmukhtar (Junaid)]

I am a newbie to Logstash and ES in general, and struggling to add S3 bucket name to logstash output.

I have a setup where we have multiple s3 buckets with RDS instance logs (1 bucket per instance) and we want to harvest the logs from the bucket.

In order to identify the instance, we need to add bucket name to the logstash output but i can't find a way to do that. I am able to add file name via [@metadata][s3][key] but what i need is the bucket name.

Can anyone help me in doing that?

Thanks,
Junaid

[magnusbaeck (Magnus Bäck)]

Aren't you setting the bucket name in the s3 input confguration (please always post your configuration when you ask a question)? Then just use add_field in the input to add the same string as a field.

[jmukhtar (Junaid)]

Details are listed below

S3 Bucket and Structure

Bucket Name: production-logs
Folder Name: Instance_A/{{Log_Types}/Log_Files ==> Log_Types is error/debug/other
Folder Name: Instance_B/{{Log_Types}/Log_Files ==> Log_Types is error/debug/other
Folder Name: Instance_C/{{Log_Types}/Log_Files ==> Log_Types is error/debug/other
Folder Name: Instance_D/{{Log_Types}/Log_Files ==> Log_Types is error/debug/other

Bucket Name: preproduction-logs
Folder Name: Instance_A/{{Log_Types}/Log_Files ==> Log_Types is error/debug/other
Folder Name: Instance_B/{{Log_Types}/Log_Files ==> Log_Types is error/debug/other
Folder Name: Instance_C/{{Log_Types}/Log_Files ==> Log_Types is error/debug/other
Folder Name: Instance_D/{{Log_Types}/Log_Files ==> Log_Types is error/debug/other

below is what i am currently doing to set the name
input {
s3 {
access_key_id => "access_key_id"
bucket => "production-logs"
region => "eu-west-1"
secret_access_key => "secret_access_key"
type => "s3"
sincedb_path => "/dev/null"
prefix => "Instance_A/error/"
add_field => { "[@metadata][buck_name]" => "Instance_A-error-logs" }
}
}
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{TZ}::@:[%{DATA:PID}]%{SPACE}:%{WORD:lvl}:%{GREEDYDATA} " }
}
mutate {
lowercase => [ "lvl" ]
remove_field => [ "message" ]
add_field => {
"hostname" => "%{[@metadata][buck_name]}"
}
}
}

This is a time consuming and error prone method and doesn't give me flexibility to add more instances to it dynamically.

If i have the option of adding "S3 bucket name" or "Folder name" dynamically then i can skip better handle the inputs

[magnusbaeck (Magnus Bäck)]

This is a time consuming and error prone method and doesn't give me flexibility to add more instances to it dynamically.

You'd obviously want to generate the configuration files.

If i have the option of adding "S3 bucket name" or "Folder name" dynamically then i can skip better handle the inputs

Yes, but that's not possible right now.