# Unable to read logs from S3 storage

**URL:** <https://discuss.elastic.co/t/unable-to-read-logs-from-s3-storage/172746>\
**Category:** Logstash\
**Created:** [March 18, 2019, 10:51am UTC](https://discuss.elastic.co/t/unable-to-read-logs-from-s3-storage/172746 "2019-03-18T10:51:04Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Meiyappan\_Kannappa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/meiyappan_kannappa/32/42219_2.png) [@Meiyappan\_Kannappa](https://discuss.elastic.co/u/Meiyappan_Kannappa)\
**Post date:** [March 18, 2019, 10:51am UTC](https://discuss.elastic.co/t/unable-to-read-logs-from-s3-storage/172746/1 "2019-03-18T10:51:04Z")

</div>

Hi,

I am trying to read logs from internal S3 storage using logstash. Below is the config. But When i try it always gives no files found in bucket. Through explorer i could see the files. Can someone help me on this. I have tried both .log and .txt files in s3 storage for testing purpose. Nothing works.  
input {  
s3 {  
"access\_key\_id" =\> "xxxxxxxxxxxxxxxxxxxxxx"  
"secret\_access\_key" =\> "xxxxxxxxxxxxxxxxxxxxx"  
"endpoint"=\>"[http://xxx.com:9020](http://xxx.com:9020)"  
"bucket" =\> "samplelogs"  
"temporary\_directory"=\>"C:/xxx/ELK/logstash-6.6.1"  
prefix=\>"/"  
add\_field =\> { source =\> gzfiles }  
type =\> "s3"

}  
}

This is logs  
Sending Logstash's logs to C:/Meiyappan/ELK/logstash-6.3.2/logs which is now configured via log4j2.properties  
[2019-03-18T15:58:00,879][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified  
[2019-03-18T15:58:01,311][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"6.3.2"}  
[2019-03-18T15:58:25,213][INFO][logstash.pipeline] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>8, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50}  
[2019-03-18T15:58:25,494][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=\>{:removed=\>, :added=\>[[http://localhost:9200/](http://localhost:9200/)]}}  
[2019-03-18T15:58:25,509][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://localhost:9200/](http://localhost:9200/), :path=\>"/"}  
[2019-03-18T15:58:25,650][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://localhost:9200/](http://localhost:9200/)"}  
[2019-03-18T15:58:25,681][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es\_version=\>6}  
[2019-03-18T15:58:25,681][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document \_type {:es\_version=\>6}  
[2019-03-18T15:58:25,713][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[//localhost:9200](https://localhost:9200)"]}  
[2019-03-18T15:58:25,728][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=\>nil}  
[2019-03-18T15:58:25,744][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "version"=\>60001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date"}, "@version"=\>{"type"=\>"keyword"}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}}  
[2019-03-18T15:58:25,759][INFO][logstash.inputs.s3] Registering s3 input {:bucket=\>"gvmslogs", :region=\>"us-east-1"}  
[2019-03-18T15:58:26,041][INFO][logstash.pipeline] Pipeline started successfully {:pipeline\_id=\>"main", :thread=\>"#\<Thread:0x6ac2a093 run\>"}  
[2019-03-18T15:58:26,103][INFO][logstash.agent] Pipelines running {:count=\>1, :running\_pipelines=\>[:main], :non\_running\_pipelines=\>}  
[2019-03-18T15:58:26,338][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}  
[2019-03-18T15:58:28,400][INFO][logstash.inputs.s3] S3 input: No files found in bucket {:prefix=\>"/"}  
[2019-03-18T15:59:27,463][INFO][logstash.inputs.s3] S3 input: No files found in bucket {:prefix=\>"/"}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 15, 2019, 11:04am UTC](https://discuss.elastic.co/t/unable-to-read-logs-from-s3-storage/172746/2 "2019-04-15T11:04:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.