Bug when attempting to use logstash-output-s3 with encryption

Elastic Stack Logstash
1

I have an s3 bucket where I have policy set like

{
    "Version": "2012-10-17",
    "Id": "PutObjPolicy",
    "Statement": [
        {
            "Sid": "DenyIncorrectEncryptionHeader",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "mybucketarn/*",
            "Condition": {
                "StringNotEquals": {
                    "s3:x-amz-server-side-encryption": "AES256"
                }
            }
        },
        {
            "Sid": "DenyUnEncryptedObjectUploads",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "mybucketarn/*",
            "Condition": {
                "Null": {
                    "s3:x-amz-server-side-encryption": "true"
                }
            }
        }
    ]
}

This requires that all files uploaded to the bucket are encrypted with AES256.
When I have my output block configured like this:

output {
   s3{
     server_side_encryption => true
     server_side_encryption_algorithm => "AES256"
     access_key_id => "mykey"
     secret_access_key => "mykey"
     region => "my-region"
     bucket => "mybucket"
     codec => "json"
      }
}

I am unable to get logstash to successfully launch, producing an error like:
[2017-06-16T13:53:38,921][ERROR][logstash.outputs.s3 ] Error validating bucket write permissions! {:message=>"Access Denied", :class=>"Aws::S3::Errors::AccessDenied"}

If I remove the policy, I am able to get encrypted files writing out to S3 just fine.

I believe this is because logstash-output-s3/lib/logstash/outputs/s3/write_bucket_permission_validator.rb attempts to create a file to check that it has write permissions. Due to the bucket policy, since the file that write_bucket_permission_validator.rb attempts to write is not encrypted, it is rejected and logstash decides that it can't output to s3.

It would be valuable if write_bucket_permission_validator and/or writable_directory_validator.rb were configured to take encryption settings in the output parameterization into account when validating write access.

I've created a github issue with the same problem here: If encryption is required on s3 bucket through bucket policy, output cannot start · Issue #146 · logstash-plugins/logstash-output-s3 · GitHub

2

We can skip this check using validate_credentials_on_root_bucket => false - not sure if this is an appropriate workaround, though? Having the validation might still be useful in the case of the encrypted buckets.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.