Hi all,
Recently we started fetching logs from s3 using logstash input s3 and it was working fine until we enabled server side encryption to the bucket.
IAM permissions look OK. I have tried with s3:* and resource:* as well, just to be sure.
here is the config used:
input {
s3 {
bucket => "XXX"
region => "eu-west-1"
role_arn => "XXX"
interval => 60
backup_to_bucket => "XXX"
delete => true
}
}
Here is the output from logstash:
Plugin: <LogStash::Inputs::S3 bucket=>"XXXX", backup_to_bucket=>"XXXX", role_arn=>"XXXX", interval=>60, id=>"XXXX", region=>"eu-west-1", delete=>true, enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"XXXX", enable_metric=>true, charset=>"UTF-8">, role_session_name=>"logstash", watch_for_new_files=>true, temporary_directory=>"/tmp/logstash", include_object_properties=>false>
Error: Access Denied
Exception: Aws::S3::Errors::AccessDenied
Stack: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.157/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.157/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.157/lib/aws-sdk-core/plugins/s3_dualstack.rb:24:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.157/lib/aws-sdk-core/plugins/s3_accelerate.rb:34:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.157/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.157/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.157/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.157/lib/seahorse/client/plugins/response_target.rb:21:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.157/lib/seahorse/client/request.rb:70:in `send_request'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.157/lib/seahorse/client/base.rb:207:in `block in define_operation_methods'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.11.157/lib/aws-sdk-resources/services/s3/object.rb:64:in `copy_from'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-3.4.1/lib/logstash/inputs/s3.rb:156:in `backup_to_bucket'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-3.4.1/lib/logstash/inputs/s3.rb:395:in `process_log'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-3.4.1/lib/logstash/inputs/s3.rb:179:in `block in process_files'
org/jruby/RubyArray.java:1734:in `each'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-3.4.1/lib/logstash/inputs/s3.rb:174:in `process_files'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-3.4.1/lib/logstash/inputs/s3.rb:118:in `block in run'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:20:in `interval'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-3.4.1/lib/logstash/inputs/s3.rb:117:in `run'
/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:409:in `inputworker'
/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:403:in `block in start_input'
[ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin
Please note, we are using 6.5.1 docker image.
I suspected it was something with the configuration as for logstash output s3, we needed to pass in server_side_encryption and ssekms_key_id etc in the config but for input s3, those options are not available. I tried to add those config in additional settings by referring to AWS API documentation but the logs said invalidation configuration key.
Later we removed s3 server side encryption and it works perfectly. After enabling it, it fails.
Any help appreciated.
Thanks!
//Raywon