S3 input plugin not working

Elastic Stack Logstash
1

Hi,

I am trying to read json files under sub directory(non-prod/abc/) in a s3 bucket(abc-bucket) using s3 input plugin with IAM roles And Below is my configuration

  input
    {
    s3 {
            bucket => "abc-bucket"
            prefix => "/non-prod/abc/drop-2017-11-13/"
            region => "us-west-2"
            codec => "json"
          } 
    }
    output {
       stdout { codec => rubydebug }
    }

In the logs i can see logstash registering to the s3 bucket and there is no error in logs. But its not reading files under drop-2017-11-13 folder. I tried changing the prefix in different ways but its not working.

In the logs i can see it found the key "S3 input: Found key " but there is no output.

curl is returning data from the file from s3 bucket.

Can some please help me to figure it out? also there will be a new folder created daily. How to read the files under drop-* dynamically?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.