Logstash S3 input plugin configuration read files from multiple directories

Elastic Stack Logstash
1

Hi All,

We have requirement which is little bit different to write configuration logstash.

Below are the files and location details

s3bucket/logs/prod/todate/prod.txt
s3bucket/logs/dev/todate/dev.txt

The input config has to read txt files from all environment directories and needs to insert in to respective indexes. Below is the one i wrote but still not clear

input {
s3 { type => "logs"
bucket => "s3bucket"
prefix => [ "logs/dev/", "logs/prod/" ]
interval => 20
access_key_id => "XXXXXXXXX"
secret_access_key => "XXXXXXXXXXXX"
}
}

As per logstash docs for s3 input prefix is not supported for regex operations but is there any other s3 input plugin available which will accept regex operations

like path => for file input

Here main agenda is read files from multiple directories with in s3 bucket and insert docs into respective index

Please provide suggestions to develop this input configuration

Thanks

2 Likes
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.