Error: uninitialized constant Aws::Client::Errors

Elastic Stack Logstash
1

I'm trying got move json files from s3 to elastic search and I get the following error:

Error: uninitialized constant Aws::Client::Errors

Here is my .conf file

input {
s3 {
bucket => "xxxxxxx"
access_key_id => "xxxxxxxxxxxxxxxxxxxxxxx"
secret_access_key => "xxxxxxxxxxxxxxxxxxx"
codec => "json"
}
}

output {
elasticsearch {
hosts => "xxxxxxxxxxxxxxxxxxxxxxxx"
ssl => false
flush_size => 250000
codec => "json"
}
}

The weird part is it works fine for a different S3 bucket. Any suggestions?

2

Having the same issue

[2016-11-03T23:22:37,565][ERROR][logstash.pipeline        ] A plugin had an unrecoverable error. Will restart this plugin.
  Plugin: <LogStash::Inputs::S3 bucket=>"my-bucket", region=>"us-east-1", prefix=>"prefix/", exclude_pattern=>"_temporary", delete=>true, backup_add_prefix=>"logstashed/", backup_to_bucket=>"my-bucket", interval=>60, type=>"message", id=>"d75fedeb20a07924471e768423feb6c6c8053d54-1", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_43d6abe1-f961-41e3-af17-aae7f6e51581", enable_metric=>true, charset=>"UTF-8">, temporary_directory=>"/tmp/logstash">
  Error: uninitialized constant Aws::Client::Errors

Using logstash from within a docker container on AWS. I do see activity on the ES instance but none of the S3 files gets deleted or moved to the backup folder.

My input config:

input {
  s3 {
    bucket => "${S3_BUCKET_NAME:my-bucket}"
    region => "${S3_REGION:us-east-1}"
    prefix => "${S3_PREFIX:prefix/}"
    exclude_pattern => "_temporary"
    delete => true
    backup_add_prefix => "${S3_BACKUP_PREFIX:logstashed/}"
    backup_to_bucket => "${S3_BUCKET_NAME:my-bucket}"
    interval => 60
    type => "message"
  }
}
3

I just noticed that the readme of the logstash-input-s3 plugin is missing some information:
The s3:PutObject action is also required in the case a backup bucket is used.

So I'd assume that is error generally has to do with insufficient permissions for the particular S3 bucket.

4

Same here. Have you managed to this problem?

5

This actually seems like an issue with the underlying JRuby

6

Just in case anyone else finds this post the same way I did, this particular issue can be a permissions problem (https://github.com/logstash-plugins/logstash-input-s3/issues/99). The way I diagnosed this was with awscli -- i did an aws s3 ls bucket/prefix/ and that worked, but an aws s3 cp s3://bucket/prefix/file /tmp/x didn't work. It tuns out that your policy needs to look something like this:

"Action": [
  "s3:DeleteObject",
  "s3:GetObject",
   "s3:ListBucket"
],
"Resource": [
   "arn:aws:s3:::bucket_name_here",
   "arn:aws:s3:::bucket_name_here/*"
]

The first resource is needed for ListBucket to work and the second is needed to allow it to download files