I am facing an error while configuring logstash to fetch data from s3 below is the configuration of logstash s3 input plugin.
input {
s3 {
access_key_id => "################"
bucket => "anil-data"
region => "###########"
secret_access_key => "###############"
prefix => "/"
type => "s3"
}
}
output {
stdout { codec => rubydebug }
}
o/p
[vagrant@slave1 logstash-6.2.4]$ ./bin/logstash -f logstash.conf
Sending Logstash's logs to /home/vagrant/fclogs/logstash-6.4.3/logs which is now configured via log4j2.properties
[2018-05-15T11:16:00,377][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/home/vagrant/logstash-6.2.4/modules/fb_apache/configuration"}
[2018-05-15T11:16:00,403][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/home/vagrant/logstash-6.2.4/modules/netflow/configuration"}
[2018-05-15T11:16:00,983][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-05-15T11:16:01,723][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.4"}
[2018-05-15T11:16:02,162][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-05-15T11:16:23,734][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-05-15T11:16:23,845][INFO ][logstash.inputs.s3 ] Registering s3 input {:bucket=>"anil-data", :region=>"us-east-2"}
[2018-05-15T11:16:24,410][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x4ada52ed run>"}
[2018-05-15T11:16:24,663][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}
[2018-05-15T11:16:28,165][ERROR][logstash.inputs.s3 ] S3 input: Unable to list objects in bucket {:prefix=>"/", :message=>"The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint."}
[2018-05-15T11:17:27,590][ERROR][logstash.inputs.s3 ] S3 input: Unable to list objects in bucket {:prefix=>"/", :message=>"The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint."}