Logstash input s3 Exception: Aws::S3::Errors::AccessDenied after enabling server side encryption

Elastic Stack Logstash
1

Hi all,

Recently we started fetching logs from s3 using logstash input s3 and it was working fine until we enabled server side encryption to the bucket.

IAM permissions look OK. I have tried with s3:* and resource:* as well, just to be sure.

here is the config used:

input {
   s3 {
       bucket            => "XXX"
       region            => "eu-west-1"
       role_arn          => "XXX"
       interval          => 60
       backup_to_bucket  => "XXX"
       delete            => true
   }
}

Here is the output from logstash:

Plugin: <LogStash::Inputs::S3 bucket=>"XXXX", backup_to_bucket=>"XXXX", role_arn=>"XXXX", interval=>60, id=>"XXXX", region=>"eu-west-1", delete=>true, enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"XXXX", enable_metric=>true, charset=>"UTF-8">, role_session_name=>"logstash", watch_for_new_files=>true, temporary_directory=>"/tmp/logstash", include_object_properties=>false>
Error: Access Denied
Exception: Aws::S3::Errors::AccessDenied
Stack: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.157/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.157/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.157/lib/aws-sdk-core/plugins/s3_dualstack.rb:24:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.157/lib/aws-sdk-core/plugins/s3_accelerate.rb:34:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.157/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.157/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.157/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.157/lib/seahorse/client/plugins/response_target.rb:21:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.157/lib/seahorse/client/request.rb:70:in `send_request'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.157/lib/seahorse/client/base.rb:207:in `block in define_operation_methods'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.11.157/lib/aws-sdk-resources/services/s3/object.rb:64:in `copy_from'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-3.4.1/lib/logstash/inputs/s3.rb:156:in `backup_to_bucket'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-3.4.1/lib/logstash/inputs/s3.rb:395:in `process_log'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-3.4.1/lib/logstash/inputs/s3.rb:179:in `block in process_files'
org/jruby/RubyArray.java:1734:in `each'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-3.4.1/lib/logstash/inputs/s3.rb:174:in `process_files'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-3.4.1/lib/logstash/inputs/s3.rb:118:in `block in run'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:20:in `interval'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-3.4.1/lib/logstash/inputs/s3.rb:117:in `run'
/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:409:in `inputworker'
/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:403:in `block in start_input'
[ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin

Please note, we are using 6.5.1 docker image.

I suspected it was something with the configuration as for logstash output s3, we needed to pass in server_side_encryption and ssekms_key_id etc in the config but for input s3, those options are not available. I tried to add those config in additional settings by referring to AWS API documentation but the logs said invalidation configuration key.

Later we removed s3 server side encryption and it works perfectly. After enabling it, it fails.

Any help appreciated.
Thanks!
//Raywon

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.