Bug when attempting to use logstash-output-s3 with encryption

I have an s3 bucket where I have policy set like

{
    "Version": "2012-10-17",
    "Id": "PutObjPolicy",
    "Statement": [
        {
            "Sid": "DenyIncorrectEncryptionHeader",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "mybucketarn/*",
            "Condition": {
                "StringNotEquals": {
                    "s3:x-amz-server-side-encryption": "AES256"
                }
            }
        },
        {
            "Sid": "DenyUnEncryptedObjectUploads",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "mybucketarn/*",
            "Condition": {
                "Null": {
                    "s3:x-amz-server-side-encryption": "true"
                }
            }
        }
    ]
}

This requires that all files uploaded to the bucket are encrypted with AES256.
When I have my output block configured like this:

output {
   s3{
     server_side_encryption => true
     server_side_encryption_algorithm => "AES256"
     access_key_id => "mykey"
     secret_access_key => "mykey"
     region => "my-region"
     bucket => "mybucket"
     codec => "json"
      }
}

I am unable to get logstash to successfully launch, producing an error like:
[2017-06-16T13:53:38,921][ERROR][logstash.outputs.s3 ] Error validating bucket write permissions! {:message=>"Access Denied", :class=>"Aws::S3::Errors::AccessDenied"}

If I remove the policy, I am able to get encrypted files writing out to S3 just fine.

I believe this is because logstash-output-s3/lib/logstash/outputs/s3/write_bucket_permission_validator.rb attempts to create a file to check that it has write permissions. Due to the bucket policy, since the file that write_bucket_permission_validator.rb attempts to write is not encrypted, it is rejected and logstash decides that it can't output to s3.

It would be valuable if write_bucket_permission_validator and/or writable_directory_validator.rb were configured to take encryption settings in the output parameterization into account when validating write access.

I've created a github issue with the same problem here: https://github.com/logstash-plugins/logstash-output-s3/issues/146

We can skip this check using validate_credentials_on_root_bucket => false - not sure if this is an appropriate workaround, though? Having the validation might still be useful in the case of the encrypted buckets.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.