S3 input plugin not working


I am trying to read json files under sub directory(non-prod/abc/) in a s3 bucket(abc-bucket) using s3 input plugin with IAM roles And Below is my configuration

    s3 {
            bucket => "abc-bucket"
            prefix => "/non-prod/abc/drop-2017-11-13/"
            region => "us-west-2"
            codec => "json"
    output {
       stdout { codec => rubydebug }

In the logs i can see logstash registering to the s3 bucket and there is no error in logs. But its not reading files under drop-2017-11-13 folder. I tried changing the prefix in different ways but its not working.

In the logs i can see it found the key "S3 input: Found key " but there is no output.

curl is returning data from the file from s3 bucket.

Can some please help me to figure it out? also there will be a new folder created daily. How to read the files under drop-* dynamically?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.