Logstash s3 input

Elastic Stack Logstash
1

Hi Team,
I am trying to read cloud trail logs from a S3 bucket, i installed cloudtrail plugin too. I can download logs from s3 bucket using "s3 cp" command so there is no connectivity issue. (Though i can't download using wget, that's our restriction, does it matter?)

While i am trying to execute logstash, i am not getting any output. Debug mode stuck on "Pushing flush onto pipeline" PFB:

how can i find out what is not working here?

/opt/logstash/bin/logstash -f logstash.conf --debug
Logstash Debug:

Starting pipeline {:id=>"main", :pipeline_workers=>1, :batch_size=>125, :batch_delay=>5, :max_inflight=>125, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"188", :method=>"start_workers"}
Pipeline main started {:file=>"logstash/agent.rb", :line=>"465", :method=>"start_pipeline"}
Pushing flush onto pipeline {:level=>:debug, :file=>"logstash/pipeline.rb", :line=>"458", :method=>"flush"}
Pushing flush onto pipeline {:level=>:debug, :file=>"logstash/pipeline.rb", :line=>"458", :method=>"flush"}
Pushing flush onto pipeline {:level=>:debug, :file=>"logstash/pipeline.rb", :line=>"458", :method=>"flush"}
Pushing flush onto pipeline {:level=>:debug, :file=>"logstash/pipeline.rb", :line=>"458", :method=>"flush"}

Configuration:
input{
s3 {
bucket => "XXXXX"
delete => false
interval => 60 # seconds
prefix => "cloudtrail/"
type => "cloudtrail"
codec => "cloudtrail"
}
}

output{
stdout{
}
}

Thanks & Regards

2

I am having the same issue. Did you find the solution? It may be due to the older files on s3.

3

Hi Jatan,
In my case the "prefilx" was not matching, so i was getting error. Try with below input and it will surely work, if network and access is not an issue. Sometimes it took 10-15 mins depending upon network bandwidth.

*Post testing you can configure other parameters like interval, delete etc. depending upon your requirements, or they will be default.

s3{
bucket => "AAA"
type => "cloudtrail"
codec => "cloudtrail"
exclude_pattern => "./CloudTrail-Digest/."
}

Regards