ElasticSearch S3 snapshot fails after writing test files

Elastic Stack Elasticsearch
1

I have configured a role in AWS to allow all operations on an s3 paths: s3:///foo/ and s3:///foo/*.
Then, I configured an S3 snapshot like:
PUT /_snapshot/s3_backup { "type": "s3", "settings": { "bucket": "<bucket>", "base_path": "foo/", "max_retries": 3 } }
Now, I try to create the snapshot using
PUT /_snapshot/s3_backup/hello_backup_1?wait_for_completion=true { "indices": "searching_index", "ignore_unavailable": "true", "include_global_state": false }
After doing so, I get the following error:
{ "error": { "root_cause": [ { "type": "amazon_s3_exception", "reason": "amazon_s3_exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: <some request id>)" } ], "type": "amazon_s3_exception", "reason": "amazon_s3_exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: <some request id>)" }, "status": 500 }
In fact when I try to create a new snapshot, with a different name, I get the same error.
However, when I look at S3, there are several test folders are created with random files, but, nothing else.
Any idea about where did I go wrong?

2

What can you see in logs?

3

com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: <requestid>), S3 Extended Request ID: <extended request id> at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1239) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:823) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:506) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:318) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3595) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3548) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:647) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:632) at org.elasticsearch.cloud.aws.blobstore.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:117) at org.elasticsearch.repositories.blobstore.BlobStoreRepository.snapshots(BlobStoreRepository.java:398) at org.elasticsearch.repositories.blobstore.BlobStoreRepository.finalizeSnapshot(BlobStoreRepository.java:376) at org.elasticsearch.snapshots.SnapshotsService$5.run(SnapshotsService.java:797) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
This was the error message

4

A common error is that you did not give access to all nodes with all the needed rights.

Could you check and compare with the documentation?

5

All nodes are given access to perform all operations on S3. So, I don't think permissions might be an issue.

6

So you have those permissions for all nodes?

{
  "Statement": [
    {
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads",
        "s3:ListBucketVersions"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::snaps.example.com"
      ]
    },
    {
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::snaps.example.com/*"
      ]
    }
  ],
  "Version": "2012-10-17"
}
7

Will this not work when I am writing snapshots inside a folder?

{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:"
],
"Resource": [
"arn:aws:s3:::snapshot/folder/",
"arn:aws:s3:::snapshot/folder/"
],
"Effect": "Allow",
"Sid": "Snapshot"
}
]
}

8

Unsure. Try what I pasted.

BTW, please format your code using </> icon.

9

Yep, the permissions that were documented worked.
Thanks a lot.