Create directories according to field in s3 bucket

Elastic Stack Logstash
1

Hi,

We are using logstash to route logs to different places like different elasticsearch clusters. Now we want to store logs in s3 bucket as well. We are using logstash-output-s3. logstash-output-s3 plugin could store all the logs in s3 bucket. but we want to create different buckets based on fields. Here is our logstash configuration

output {

  s3{

    region => "us-east-1"

    bucket => "mycompany-qa-application-logs"

    prefix => "app/%{[attribute][APPLICATION]}-%{[attribute][ENVIRONMENT]}-%{[logType]}-logs"

    size_file => 2146304

    time_file => 5

    canned_acl => "private"

  }

}

it should create app-qa-error-logs. but it created %{[attribute][APPLICATION]}-%{[attribute][ENVIRONMENT]}-%{[logType]}-logs under app directory.

Any help will be much appreciated.

Cheers
Ferdous Shibly

2

How is the attribute field supposed to be populated?

3

attribute field is already in the logs. We use json filter to separate out. We used

prefix => "app/%{[attribute][APPLICATION]}-%{[attribute][ENVIRONMENT]}-%{[logType]}-logs"

which is creating the directories accordingly. Now we are facing a new challenge. Some of our applications doesn't have logType field in their logs. How can we send the logs to error directory if the logType is missing, for example, app1 doesn't have logType, so the logs will be sent to app1-error-logs directory?

Thanks for your help.

4
1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.