# Logstash is not pulling data from s3 bucket **URL:** \ **Category:** Logstash\ **Created:** [January 17, 2020, 1:47pm UTC](https://discuss.elastic.co/t/logstash-is-not-pulling-data-from-s3-bucket/215470 "2020-01-17T13:47:38Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![Sreekanth3](https://avatars.discourse-cdn.com/v4/letter/s/4491bb/32.png) [@Sreekanth3](https://discuss.elastic.co/u/Sreekanth3)\ **Post date:** [January 17, 2020, 1:47pm UTC](https://discuss.elastic.co/t/logstash-is-not-pulling-data-from-s3-bucket/215470/1 "2020-01-17T13:47:38Z")
Hi , This is my logstash conf input { s3 { access\_key\_id =\> "xxxx" secret\_access\_key =\> "xxx" #bucket =\> "mybucketname/2020/01/16" bucket =\> "sftlcloudtrail" sincedb\_path =\> "/tmp/last-s3-file-s3-access-logs" additional\_settings =\> { force\_path\_style =\> true follow\_redirects =\> false } } } output { elasticsearch { hosts =\> ["[http://192.168.1.72:9200](http://192.168.1.72:9200)","[http://192.168.1.62:9200](http://192.168.1.62:9200)"] index =\> "s3flowlogs-16012020" } stdout { codec =\> rubydebug } } The logstash is starting fine and the logs from the bucket are not pulled to the elasticsearch. I can't figure is there any error in the conf or what ? > Sending Logstash logs to /opt/logstash-7.4.0/logs which is now configured via log4j2.properties > [2020-01-17T19:07:41,613][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified > [2020-01-17T19:07:41,632][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"7.4.0"} > [2020-01-17T19:07:51,180][INFO][org.reflections.Reflections] Reflections took 278 ms to scan 1 urls, producing 20 keys and 40 values > [2020-01-17T19:08:52,140][INFO][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=\>{:removed=\>, :added=\>[[http://192.168.1.72:9200/](http://192.168.1.72:9200/), [http://192.168.1.62:9200/](http://192.168.1.62:9200/)]}} > [2020-01-17T19:08:52,811][WARN][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=\>"[http://192.168.1.72:9200/](http://192.168.1.72:9200/)"} > [2020-01-17T19:08:52,899][INFO][logstash.outputs.elasticsearch][main] ES Output version determined {:es\_version=\>7} > [2020-01-17T19:08:52,906][WARN][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document \_type {:es\_version=\>7} > [2020-01-17T19:08:52,916][WARN][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=\>"[http://192.168.1.62:9200/](http://192.168.1.62:9200/)"} > [2020-01-17T19:08:52,971][INFO][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=\>"LogStash::Outputs::Elasticsearch", :hosts=\>["[http://192.168.1.72:9200](http://192.168.1.72:9200)", "[http://192.168.1.62:9200](http://192.168.1.62:9200)"]} > [2020-01-17T19:08:53,330][INFO][logstash.outputs.elasticsearch][main] Using default mapping template > [2020-01-17T19:08:53,629][INFO][logstash.outputs.elasticsearch][main] Attempting to install template {:manage\_template=\>{"index\_patterns"=\>"logstash-_", "version"=\>60001, "settings"=\>{"index.refresh\_interval"=\>"5s", "number\_of\_shards"=\>1}, "mappings"=\>{"dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date"}, "@version"=\>{"type"=\>"keyword"}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}} > [2020-01-17T19:08:53,723][WARN][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster\_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team. > [2020-01-17T19:08:53,786][INFO][logstash.javapipeline][main] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50, "pipeline.max\_inflight"=\>500, :thread=\>"#\"} > [2020-01-17T19:08:54,036][INFO][logstash.inputs.s3][main] Registering s3 input {:bucket=\>"xxxx", :region=\>"us-east-1"} > [2020-01-17T19:08:56,204][INFO][logstash.javapipeline][main] Pipeline started {"pipeline.id"=\>"main"} > [2020-01-17T19:08:56,815][INFO][logstash.agent] Pipelines running {:count=\>1, :running\_pipelines=\>[:main], :non\_running\_pipelines=\>} > [2020-01-17T19:08:59,626][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9601} > [2020-01-17T19:09:13,201][INFO][logstash.inputs.s3][main] Using the provided sincedb\_path {:sincedb\_path=\>"/tmp/last-s3-file-s3-access-logs"} ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 14, 2020, 1:51pm UTC](https://discuss.elastic.co/t/logstash-is-not-pulling-data-from-s3-bucket/215470/2 "2020-02-14T13:51:28Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.