Elasticsearch S3 Snapshot Repository - Is s3:DeleteObject Mandatory for S3 Repository and SLM?

Elastic Stack Elasticsearch
1

Hi Team,

We are configuring Elasticsearch 9.1.3 snapshots to an AWS S3 bucket using the repository-s3 plugin and would like to clarify whether s3:DeleteObject permission is mandatory.

Environment

  • Elasticsearch Version: 9.1.3
  • Deployment: AWS EC2
  • Authentication: IAM Role (Instance Profile)
  • Snapshot Repository: S3
  • Snapshot Lifecycle Management (SLM): Planned for daily automated backups

IAM Permissions

The IAM Role currently has the following permissions:

  • s3:ListBucket
  • s3:GetBucketLocation
  • s3:GetObject
  • s3:PutObject

The client is not willing to provide:

  • s3:DeleteObject
    due to their organization's security policy.

Current Behaviour

Elasticsearch is able to:

  • Authenticate successfully using the IAM Role.
  • Connect to the S3 bucket.
  • Upload temporary verification files (master.dat, data-*.dat) into the configured base_path.

However, repository verification fails with:
repository_verification_exception
[s3_repository] cannot delete test data at [ELK_BACKUP]

The detailed error indicates:
AccessDenied
is not authorized to perform:
s3:DeleteObject

image
image1321×564 127 KB

The temporary verification files remain in the S3 bucket because Elasticsearch cannot delete them.

Questions

  1. Is s3:DeleteObject mandatory for configuring an S3 snapshot repository?

  2. Is there any supported way to configure the repository without granting s3:DeleteObject permission?

  3. Can snapshots be created successfully if repository verification cannot delete temporary verification files?

  4. Is there any Elasticsearch setting to skip repository cleanup or disable deletion of temporary verification files?

  5. If s3:DeleteObject is not granted, what functionality will be affected?
    Repository verification
    Snapshot creation
    Snapshot restore
    Snapshot Lifecycle Management (SLM) retention
    Repository cleanup

Our objective is to enable daily automated S3 backups while complying with the client's security policy, which currently allows only:

  • s3:ListBucket
  • s3:GetBucketLocation
  • s3:GetObject
  • s3:PutObject

Any guidance or best practices for this scenario would be greatly appreciated.

2

s3:DeleteObject is required for normal Elasticsearch S3 snapshot repository operation. Repository verification needs to create and delete test files, and SLM also needs delete access to remove expired snapshots. There’s no supported option to disable this cleanup, so the best approach is to allow s3:DeleteObject only on the specific backup bucket/prefix.

3

See also Support immutable repository when using snapshot and restore along with point in time recovery · Issue #118187 · elastic/elasticsearch · GitHub.

If you forbid DeleteObject then some things might actually even appear to work today. Definitely not supported tho, and it is not advisable to trust your snapshots to an unsupported configuration. So yes, in that sense, this permission is mandatory.

The client is not willing to provide:

  • s3:DeleteObject
    due to their organization's security policy.

This sounds like security theater to me. If an attacker can PutObject then they can modify the contents of objects, rendering snapshots inaccessible (or even just subtly altering the data they contain) even if they cannot DeleteObject. The proper approach is to enable versioning, which avoids all these problems and eliminates any security concerns around DeleteObject calls too.

4

What a cool expression, which is new to me, tho the concept is very far from new.

Every day's a school day!

5

Does this IAM role has permissions to other buckets as well?

Not willing to provide the s3:DeleteObject permission would make sense if this role has access to multiple buckets, like buckets used to store data that are being ingested in the cluster.

In this case I think you could use an access_key/access_secret with the required permissions for only the snapshot bucket.

6

I think it'd be preferable to set up the IAM role with finer-grained permissions, forbidding all write access to the ingested data source bucket/path while allowing all the required permissions on the snapshot bucket/path. Managing fixed keys is kinda painful, you have to arrange to rotate them, and they're much easier to leak than the credentials used for an IAM role.