Logstash output to AWS S3 bucket not working

Elastic Stack Logstash
1

Hi,
I've tried a couple different ways to output data to an S3 bucket on Amazon. I followed the documentation for output plug-ins to s3 but the function is not working. any suggestions please?
filter { }
output {
stdout {
codec => dots
}
elasticsearch {
hosts => "http://localhost:9200"
index => "twitter"
document_type => "tweets"
template => "D:/Downloads/datasets/iot/twitter_template.json"
template_name => "twitter_example"
template_overwrite => true
}
file {
path => "D:/Downloads/datasets/twitter_iot/twitter.csv"
}
s3 {
credentials => ["accessKeyID", "SecretKeyID"]
access_key_id => "accessKeyID"
secret_access_key => "SecretKeyID"
region => “us-west-2”
bucket => “s3://mybucket/path”
}
}

2

Have you checked the Logstash logs for clues?

3

I'm trying to do a similar thing, but using environment variables for the access & secret keys. I get this error in response:
"{:timestamp=>"2016-05-09T15:58:41.247000-0700", :message=>"S3: AWS error", :error=>#AWS::S3::Errors::InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.>, :level=>:error}"

the entry in my output.conf file is:
output {
s3 {
access_key_id => "${AWS_ACCESS_KEY}"
secret_access_key => "${AWS_SECRET_KEY}"
}
}

is the s3 output plugin not handling environment variables?

4

looks like I need to add the "--allow-env" flag to running logstash to make this work. I'll try adding the flag and let you know.

5

Ok, got the env vars working, now have this error:
{:timestamp=>"2016-05-09T16:17:29.026000-0700", :message=>"S3: AWS error", :error=>#AWS::S3::Errors::PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.>, :level=>:error}

what does this mean and how is it rectified?

6

Ok. Fixed this by removing the "region" option from s3 config. Now have to figure out why my log files on s3 are filled with this line:
%{host} %{message}

7

Now have to figure out why my log files on s3 are filled with this line:
%{host} %{message}

Please start a new thread for that question, if necessary.