S3 output requires PutObject on * Resources

[Theodore_Cowan]

Posting this so somebody else may avoid spending hours trying to debug the s3 output plugin.

The plugin will throw permissions errors if you do not grant PutObject permissions on bucket resources.

[2017-10-11T02:53:41,488][ERROR][logstash.outputs.s3 ] Error validating bucket write permissions! {:message=>"Access Denied", :class=>" Aws::S3::Errors::AccessDenied",

The IAM policy required is:

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::mybucket*"
            ]
        }
    ]
}

[mujtabahussain]

Maybe mark your post as RESOLVED as well so people can see this is a solved issue

and thanks

[mujtabahussain]

Also, just a heads up, Resource section in CFN should be as restrictive as possible. So instead of * give it the buckets that you know need the s3:PutObject permission.

[Theodore_Cowan]

That's the exact problem. Specifying a bucket ARN in Resources does not work. I tested the IAM policy limiting access to the specific bucket using the AWS CLI, and it worked as expected. However, it did not work with logstash.

In addition, I was only able to make this work with a bucket in the us-east-1 region (???).

@mujtabahussain I challenge you! Setup Logstash s3 output with logstash-5.4.3-1 to a bucket in us-west-2 region with a IAM user that is appropriately scoped to a single bucket. It will not work. And if it does work, I beg you to show me your configuration.

[mujtabahussain]

@Theodore_Cowan

That's the exact problem. Specifying a bucket ARN in Resources does not work. I tested the IAM policy limiting access to the specific bucket using the AWS CLI, and it worked as expected. However, it did not work with logstash.

My bad. I should read more carefully

In addition, I was only able to make this work with a bucket in the us-east-1 region (???).

That seems very strange.

I challenge you! Setup Logstash s3 output with logstash-5.4.3-1 to a bucket in us-west-2 region with a IAM user that is appropriately scoped to a single bucket. It will not work. And if it does work, I beg you to show me your configuration.

haha ...I shall definitely try

[Theodore_Cowan]

I've updated my post. I apologize for my temporary insanity. But it did seem that I had a mysterious problem with the s3 output plugin when I really had an unknown VPC Routing Endpoint policy preventing me from performing a PutObject on my bucket.

Thanks for the responses @mujtabahussain. I have it working now.

[mujtabahussain]

All good