Logstash S3 output requires bucket object permissions at the root of the bucket, regardless of prefix config

Elastic Stack Logstash
1

Logstash requires write access to the root of the bucket, regardless of the prefix defined in config.

There are a few issues to note with this behavior:

  • Logstash should be isolating its work into the defined prefix path in config, but it's not.

  • I'm forced to give logstash write access to the root of the bucket! ("Resource": "arn:aws:s3:::my-log-bucket/*") regardless of the prefix setting.

  • After starting logstash it creates test-write files in the root of the bucket, not in the defined prefix path. The logs themselves go in the prefix, these test files should too.

  • Even though the "root" check can be disabled, (S3 output plugin | Logstash Reference [7.16] | Elastic) however I don't want to disable permission checking overall.

Bucket contents:

logstash-programmatic-access-test-object-2022-01-10 21:44:46 +0000
logstash-programmatic-access-test-object-2022-01-10 21:58:05 +0000
logstash-programmatic-access-test-object-2022-01-10 21:59:34 +0000
logstash-programmatic-access-test-object-2022-01-10 22:01:27 +0000
logstash-programmatic-access-test-object-2022-01-10 22:04:12 +0000

logstash/...

Error thrown:

LogStash::ConfigurationError: Logstash must have the privileges to write to root bucket `my-log-bucket`, check your credentials or your permissions.>

Below is the config, that "should" work as-is, but since Logstash isn't respecting the prefix, the above error is shown and logstash doesn't start.

Output config looks like this:

output {
  s3 {
    region => "us-east-1"
    bucket => "my-log-bucket"
    prefix => "logstash/"
    encoding => "gzip"
    server_side_encryption => true
    server_side_encryption_algorithm => "aws:kms"
    size_file => 5242880
    time_file => 30
  }
}

An access policy such as this should suffice, but it doesn't. The entire root object needs to be granted, not just the prefix.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::my-log-bucket"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::my-log-bucket/logstash/*"
        }
    ]
}

Granting it write access to "arn:aws:s3:::my-log-bucket/*" puts a band-aid on the problem but exposes more permissions than should be needed for any given service.

This setting S3 output plugin | Logstash Reference [7.16] | Elastic appears to just disable the check completely which is not ideal.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.