S3 No files found in bucket

Elastic Stack Logstash
1

Hi,
I'm struggling with logstash conf to read from S3 bucket which has the latest date, example structure in S3 bucket as follows:

  • apachelogs/web/2021-08-22/
  • apachelogs/web/2021-08-23/
  • apachelogs/web/2021-08-24/

Hence, my logstash input as follow

input {
    s3 {
        bucket => "apachelogs"
        prefix => "web/%{+YYYY-MM-dd}/"
        region => "ap-southeast-1"
        access_key_id => "xxxx"
        secret_access_key => "xxxx"
        sincedb_path => "/tmp/s3.sincedb"
        backup_to_dir => "/tmp/logstashed/"
        type => "apache"
    }
}

However, I kept getting
[INFO ] 2021-08-24 18:04:06.804 [[main]<s3] s3 - No files found in bucket {:prefix=>"apachelogs/web/%{+YYYY-MM-dd}/"} and interestingly with hardcoded date, it worked -> prefix=>"apachelogs/web/2021-08-24/"}

Appreciate some help here. Thanks.

2

A sprintf reference to a date uses the value of [@timestamp] from an event. This cannot be used in an input plugin since there are no events when the input is being configured. It will, as you have found, use the literal value.

3

Hi Badger, thanks for the response, so with this, is there any workaround i can do?

4

Start logstash once a day and pass the date using an environment variable.

5

Perfect, thanks a bunch!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.