Dynamic Bucket Names or Directories in AWS S3 Output

Elastic Stack Logstash
1

Hi,

I am doing some tests on storing data on AWS S3. I have read the documents and couldn't find a dynamic bucket name or directory option like Elasticsearch output provides in index name. Is there a way to do this?

Thank you

2

You can use exactly the same method for that output.

3

Can you provide an example for it? I tried it several ways and it doesn't work. I am using 1.5.0 version of Logstash.

4

Providing what you have tried would be useful :slight_smile:

5

I've been trying to get this to work also. My dev set up works just fine with a config file of

input {
  file {
    path => "/srv/log/app/server/*.log"
  }
}

filter {
  grok {
    match => ["path","%{GREEDYDATA:folder}/%{GREEDYDATA:filename}\.log"]
  }
}

output {
  s3 {
    bucket => "test"
    prefix => "test/"
    size_file => 2048
    time_file => 5
    canned_acl => "private"
    codec => rubydebug
  }
  stdout { codec => rubydebug }
}

I'd like to be able to use a wild card in the prefix to do something like prefix => "test/%{folder}", which the docs made me think might work, but that doesn't seem to work.

6

I'd like to be able to use a wild card in the prefix to do something like prefix => "test/%{folder}", which the docs made me think might work, but that doesn't seem to work.

Sorry, that won't work since %{varname} interpolation doesn't take place for the prefix parameter's value (the highlighted line indicates that we're using the raw parameter value, @prefix, instead of event.sprintf(@prefix)):

However, fixing this is probably not entirely simple since it would mean that the output file could potentially change between every single message received by the output.

7

Thanks, Now that I think about it I see how that would be a huge problem.

8

How about the other way around? Is logstash able to support dynamic bucket names in s3 input similar to how it does for file inputs?

e.g.
input {
s3 {
bucket => "logbucket"
prefix => "logs/*/2015/01/01/"
}
}

I want to use a grok filter on the s3 prefix to add fields to my log entries

grok {
match => [ "prefix", "logs/%{GREEDYDATA:projectName/2015/01/01/" ]
}

9

Since my set of prefix's was known, I worked around this by putting if conditions and hard coded prefixes:

output {
  # Until s3 output supports variables in prefix
  if [fields][host] == "foohost" {
     s3 {
       access_key_id => "<your access>"
       secret_access_key => "<your secret>"
       bucket => "host-logs"
       time_file => 60
       prefix => "foohost"
    }
  }
  if [fields][host] == "barhost" {
     s3 {
       access_key_id => "<your access>"
       secret_access_key => "<your secret>"
       bucket => "host-logs"
       time_file => 60
       prefix => "barhost"
    }
  }
  ...
}
10

I am currently trying to do something similar with my s3 output. I have input configurations coming from multiple file locations, and depending on the directory the logs come from I am set the "type" field to a specific value. Is there no way to use a field value as a prefix or part of the tags? I need to make different s3 bucket objects based on the type otherwise they all get written to the same object in s3.

The work-around would be to use if conditions, but it would be simpler to be able to use references in the s3 output.