Require example/tutorial for Logstash S3 Input Plugin

Elastic Stack Logstash
1

I am facing some issues in configuring the logstash s3 input plugin. I tried searching for a tutorial for this plugin but couldn't find any. Could you please provide a link to the tutorial/examples for this plugin if there is one. Otherwise, could you please upload a tutorial/example for the mentioned plugin. I require it for Logstash v6.0 and above.

Thanks,
Anil

2

You'll probably have better luck asking concrete questions about the problems you're having.

3

Below is my logstash configuration

input {
s3 {
access_key_id => "##############"
bucket => "anil-data"
region => "###########"
secret_access_key => "############"
prefix => "/data_log/audit/"
type => "s3"
}
}
output {
stdout { codec => rubydebug }
}

And I am getting the following output

[vagrant@slave1 logstash-6.2.4]$ ./bin/logstash -f logstash.conf
Sending Logstash's logs to /home/vagrant/fclogs/logstash-6.4.3/logs which is now configured via log4j2.properties
[2018-05-15T11:38:46,005][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/home/vagrant/logstash-6.2.4/modules/fb_apache/configuration"}
[2018-05-15T11:38:46,038][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/home/vagrant/logstash-6.2.4/modules/netflow/configuration"}
[2018-05-15T11:38:46,631][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-05-15T11:38:47,305][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.4"}
[2018-05-15T11:38:47,744][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-05-15T11:39:10,034][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-05-15T11:39:10,173][INFO ][logstash.inputs.s3 ] Registering s3 input {:bucket=>"s3-.amazonaws.com", :region=>"us-west-2"}
[2018-05-15T11:39:10,647][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x2ab014a5 run>"}
[2018-05-15T11:39:10,896][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}
[2018-05-15T12:07:59,433][ERROR][logstash.inputs.s3 ] S3 input: Unable to list objects in bucket {:prefix=>"/data_log/audit/", :message=>"The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint."}

I am not sure what to pass as endpoint.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.