# AWS S3 bucket logs to Elastic search service (AWS) through log stash
**URL:** https://discuss.elastic.co/t/aws-s3-bucket-logs-to-elastic-search-service-aws-through-log-stash/199737
**Category:** Logstash
**Created:** [September 16, 2019, 11:21pm UTC](https://discuss.elastic.co/t/aws-s3-bucket-logs-to-elastic-search-service-aws-through-log-stash/199737 "2019-09-16T23:21:52Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@sk545](https://discuss.elastic.co/u/sk545)
#### Post date: [September 16, 2019, 11:21pm UTC](https://discuss.elastic.co/t/aws-s3-bucket-logs-to-elastic-search-service-aws-through-log-stash/199737/1 "2019-09-16T23:21:52Z")
- I set up log stash on EC2 Linux (AWS) and Elastic search service on AWS
- and i have created one separate file (with input and output) under /etc/logstash/conf.d path
- but I am not getting logs in Kibana
- and connection has been established between two nodes successfully
any suggestion to check where I am missing this?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 17, 2019, 4:52am UTC](https://discuss.elastic.co/t/aws-s3-bucket-logs-to-elastic-search-service-aws-through-log-stash/199737/2 "2019-09-17T04:52:29Z")
It would help if you showed us your config.
---
### Author:  [@sk545](https://discuss.elastic.co/u/sk545)
#### Post date: [September 18, 2019, 1:49am UTC](https://discuss.elastic.co/t/aws-s3-bucket-logs-to-elastic-search-service-aws-through-log-stash/199737/3 "2019-09-18T01:49:00Z")
my config file :
input {
s3 {
#"access\_key\_id" =\> "your\_access\_key\_id"
#"secret\_access\_key" =\> "your\_secret\_access\_key"
"region" =\> "us-east"
"bucket" =\> "my bucket\_name"
#"prefix" =\> "Logs"
#"interval" =\> "10"
#"additional\_settings" =\> {
#"force\_path\_style" =\> true
#"follow\_redirects" =\> false
}
}
}
output {
elasticsearch {
hosts =\> ["[https://xxxxxxx.es.amazonaws.com:443](https://xxxxxxx.es.amazonaws.com:443)"]
index =\> "logs-test"
#user =\> "elastic"
#password =\> "changeme"
}
}
i am getting error from /var/log/logstash/logstash-plain.log
[2019-09-18T01:39:32,179][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, input, filter, output at line 14, column 1 (byte 368) after ", :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in`compile\_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in`map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in`initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in`initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline\_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in`block in converge\_state
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 18, 2019, 5:19am UTC](https://discuss.elastic.co/t/aws-s3-bucket-logs-to-elastic-search-service-aws-through-log-stash/199737/4 "2019-09-18T05:19:20Z")
This error is because you have a closing curly brace related to the additional settings block at the end of the input section that you have not commented out. I have never used AES ES, but recall seeing some users needing to use the `amazon_es` output plugin as well.
---
### Author:  [@sk545](https://discuss.elastic.co/u/sk545)
#### Post date: [September 18, 2019, 9:18pm UTC](https://discuss.elastic.co/t/aws-s3-bucket-logs-to-elastic-search-service-aws-through-log-stash/199737/5 "2019-09-18T21:18:53Z")
i have tweaked my config file a bit and it looks like below
input {
s3 {
#"access\_key\_id" =\> "your\_access\_key\_id"
#"secret\_access\_key" =\> "your\_secret\_access\_key"
"region" =\> "us-east"
"bucket" =\> "xxxxxxxx-log"
#"prefix" =\> "Logs"
#"interval" =\> "10"
}
}
output {
elasticsearch {
hosts =\> ["https:xxxxxxx.amazonaws.com:443"]
index =\> "s3logs-\*"
#user =\> "elastic"
#password =\> "changeme"
}
}
at this time i am not getting any errors except below
[2019-09-18T19:49:45,476][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"7.3.2"}
[2019-09-18T19:50:25,895][INFO][org.reflections.Reflections] Reflections took 1881 ms to scan 1 urls, producing 19 keys and 39 values
[2019-09-18T20:35:07,880][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"7.3.2"}
[2019-09-18T20:35:09,864][INFO][org.reflections.Reflections] Reflections took 105 ms to scan 1 urls, producing 19 keys and 39 values
[2019-09-18T20:50:13,114][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"7.3.2"}
[2019-09-18T20:50:16,230][INFO][org.reflections.Reflections] Reflections took 109 ms to scan 1 urls, producing 19 keys and 39 values
but still i have no idea what i am missing here !!
---
### Author:  [@sk545](https://discuss.elastic.co/u/sk545)
#### Post date: [September 18, 2019, 9:29pm UTC](https://discuss.elastic.co/t/aws-s3-bucket-logs-to-elastic-search-service-aws-through-log-stash/199737/6 "2019-09-18T21:29:41Z")
I am directly using S3 plug-in in log stash
Do i need to enable S3 I/P and O/P plug in before utilizing it.
---
### Author:  [@sk545](https://discuss.elastic.co/u/sk545)
#### Post date: [September 26, 2019, 6:35pm UTC](https://discuss.elastic.co/t/aws-s3-bucket-logs-to-elastic-search-service-aws-through-log-stash/199737/7 "2019-09-26T18:35:49Z")
finally i ended up with positive result , now i successfully shipped S3 bucket logs through Logasth to AWS ES service
my input and o/p config files as below:
**I/P config file:**
input {
s3 {
bucket =\> ""
region =\> "us-east-1"
prefix =\> "\"
}
}
**O/P config file:**
output {
amazon\_es {
hosts =\> ["[vpcxxxxxxx.es.amazonaws.com](http://vpcxxxxxxx.es.amazonaws.com)"]
index =\> "test"
aws\_access\_key\_id =\> ''
aws\_secret\_access\_key =\> ''
}
}
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 24, 2019, 6:35pm UTC](https://discuss.elastic.co/t/aws-s3-bucket-logs-to-elastic-search-service-aws-through-log-stash/199737/8 "2019-10-24T18:35:52Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.