# Logstash s3 input

**URL:** <https://discuss.elastic.co/t/logstash-s3-input/54984>\
**Category:** Logstash\
**Created:** [July 7, 2016, 6:40pm UTC](https://discuss.elastic.co/t/logstash-s3-input/54984 "2016-07-07T18:40:32Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\
**Post date:** [July 7, 2016, 6:40pm UTC](https://discuss.elastic.co/t/logstash-s3-input/54984/1 "2016-07-07T18:40:32Z")

</div>

Hi Team,  
I am trying to read cloud trail logs from a S3 bucket, i installed cloudtrail plugin too. I can download logs from s3 bucket using "s3 cp" command so there is no connectivity issue. (Though i can't download using wget, that's our restriction, does it matter?)

While i am trying to execute logstash, i am not getting any output. Debug mode stuck on "Pushing flush onto pipeline" PFB:

how can i find out what is not working here?

/opt/logstash/bin/logstash -f logstash.conf --debug  
Logstash Debug:

Starting pipeline {:id=\>"main", :pipeline\_workers=\>1, :batch\_size=\>125, :batch\_delay=\>5, :max\_inflight=\>125, :level=\>:info, :file=\>"logstash/pipeline.rb", :line=\>"188", :method=\>"start\_workers"}  
Pipeline main started {:file=\>"logstash/agent.rb", :line=\>"465", :method=\>"start\_pipeline"}  
Pushing flush onto pipeline {:level=\>:debug, :file=\>"logstash/pipeline.rb", :line=\>"458", :method=\>"flush"}  
Pushing flush onto pipeline {:level=\>:debug, :file=\>"logstash/pipeline.rb", :line=\>"458", :method=\>"flush"}  
Pushing flush onto pipeline {:level=\>:debug, :file=\>"logstash/pipeline.rb", :line=\>"458", :method=\>"flush"}  
Pushing flush onto pipeline {:level=\>:debug, :file=\>"logstash/pipeline.rb", :line=\>"458", :method=\>"flush"}

Configuration:  
input{  
s3 {  
bucket =\> "XXXXX"  
delete =\> false  
interval =\> 60 # seconds  
prefix =\> "cloudtrail/"  
type =\> "cloudtrail"  
codec =\> "cloudtrail"  
}  
}

output{  
stdout{  
}  
}

Thanks & Regards

---

<div class="post-metadata">

**Author:** ![jatanpatel92](https://avatars.discourse-cdn.com/v4/letter/j/d07c76/32.png) [@jatanpatel92](https://discuss.elastic.co/u/jatanpatel92)\
**Post date:** [August 1, 2016, 6:01pm UTC](https://discuss.elastic.co/t/logstash-s3-input/54984/2 "2016-08-01T18:01:01Z")

</div>

I am having the same issue. Did you find the solution? It may be due to the older files on s3.

---

<div class="post-metadata">

**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\
**Post date:** [August 1, 2016, 6:36pm UTC](https://discuss.elastic.co/t/logstash-s3-input/54984/3 "2016-08-01T18:36:04Z")

</div>

Hi Jatan,  
In my case the "prefilx" was not matching, so i was getting error. Try with below input and it will surely work, if network and access is not an issue. Sometimes it took 10-15 mins depending upon network bandwidth.

\*Post testing you can configure other parameters like interval, delete etc. depending upon your requirements, or they will be default.

s3{  
bucket =\> "AAA"  
type =\> "cloudtrail"  
codec =\> "cloudtrail"  
exclude\_pattern =\> "._/CloudTrail-Digest/._"  
}

Regards

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:45am UTC](https://discuss.elastic.co/t/logstash-s3-input/54984/4 "2017-07-06T04:45:32Z")

</div>