Is there a bug in elasticsearch 6.5.0 concerning s3 snapshot repositories

Elastic Stack Elasticsearch
1

Hi All,

This is a single node cluster installation running elasticsearch 6.5.0.
I noticed that after the set up of the aws access key and secret key in the keystore

P.S. I have tested my access key and secret key with cyberduck and I can connect, read, write objects, etc. in the s3 bucket.

Request:

PUT _snapshot/es5_aws_test_backup
{
  "type": "s3",
  "settings": {
    "bucket": "metrika-aws-backups"
  }
}

Response:

{
      "error": {
        "root_cause": [
          {
            "type": "repository_exception",
            "reason": "[es5_aws_test_backup] cannot create blob store"
          }
        ],
        "type": "repository_exception",
        "reason": "[es5_aws_test_backup] cannot create blob store",
        "caused_by": {
          "type": "illegal_argument_exception",
          "reason": "you do not have permissions to access the bucket [test-aws-backups]",
          "caused_by": {
            "type": "amazon_s3_exception",
            "reason": "Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: EA157F3D99CF2FF1; S3 Extended Request ID: YpJeteKUoY6I5vSTp7gEVVa1DXjwbHFVIhhYJ80/F2Vu2kIHKN/wfreoGhNZuGWNpK/C+h9i1wU=)"
          }
        }
      },
      "status": 500
    }

Server logs:

[2020-08-24T23:27:27,288][WARN ][r.suppressed             ] [node-5] path: /_snapshot/es5_aws_test_backup, params: {pretty=, repository=es5_aws_test_backup}
    org.elasticsearch.repositories.RepositoryException: [es5_aws_test_backup] cannot create blob store
            at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:336) ~[elasticsearch-6.5.0.jar:6.5.0]
            at org.elasticsearch.repositories.s3.S3Repository.blobStore(S3Repository.java:258) ~[?:?]
            at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:635) ~[elasticsearch-6.5.0.jar:6.5.0]
            at org.elasticsearch.repositories.RepositoriesService.lambda$verifyRepository$2(RepositoriesService.java:218) ~[elasticsearch-6.5.0.jar:6.5.0]
            at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) [elasticsearch-6.5.0.jar:6.5.0]
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_231]
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_231]
            at java.lang.Thread.run(Thread.java:748) [?:1.8.0_231]
    Caused by: java.lang.IllegalArgumentException: you do not have permissions to access the bucket [test-aws-backups]
            at org.elasticsearch.repositories.s3.S3BlobStore.lambda$new$0(S3BlobStore.java:79) ~[?:?]
            at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:57) ~[?:?]
            at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_231]
            at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:56) ~[?:?]
            at org.elasticsearch.repositories.s3.S3BlobStore.<init>(S3BlobStore.java:72) ~[?:?]
            at org.elasticsearch.repositories.s3.S3Repository.createBlobStore(S3Repository.java:251) ~[?:?]
            at org.elasticsearch.repositories.s3.S3Repository.createBlobStore(S3Repository.java:53) ~[?:?]
            at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:332) ~[elasticsearch-6.5.0.jar:6.5.0]
            ... 7 more
    Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 0F6E6291D55D6322; S3 Extended Request ID: 6+O2pdgTQh+SYssTdRzisOlYAjHt3miNgA7SBL1eVm12ULRHA31Nw0TSoSqNtgcU6AzgvGdka58=)
            at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1658) ~[?:?]
            at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1322) ~[?:?]
            at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1072) ~[?:?]
            at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:745) ~[?:?]
            at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719) ~[?:?]
            at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701) ~[?:?]
            at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669) ~[?:?]
            at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651) ~[?:?]
            at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515) ~[?:?]
            at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4443) ~[?:?]
            at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4390) ~[?:?]
            at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1354) ~[?:?]
            at org.elasticsearch.repositories.s3.S3BlobStore.lambda$new$0(S3BlobStore.java:74) ~[?:?]
            at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:57) ~[?:?]
            at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_231]
            at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:56) ~[?:?]
            at org.elasticsearch.repositories.s3.S3BlobStore.<init>(S3BlobStore.java:72) ~[?:?]
            at org.elasticsearch.repositories.s3.S3Repository.createBlobStore(S3Repository.java:251) ~[?:?]
            at org.elasticsearch.repositories.s3.S3Repository.createBlobStore(S3Repository.java:53) ~[?:?]
            at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:332) ~[elasticsearch-6.5.0.jar:6.5.0]
            ... 7 more
2

Have you checked the auth permissions against the bucket?

3

Thank you for your prompt reply!
Can you please elaborate? How can I check that (except of trying to independently connect with another client)?

4

From CyberDuck I can see:
ACL
Grantee Permission
Admin FULL_CONTROL
Log Delivery READ_ACP
Log Delivery WRITE

Should I enforce a more specific policy?
I understand that I should be able to check if the user which belongs to my account is able to access the bucket and list, put, delete objects.
If I can do all these actions from a client why elasticsearch cannot do the same thing?

5

I made a small python3 script using boto3 and I am able using the same credentials to access and write an object in the bucket.
I cannot understand why elasticsearch using these credentials fails ... Is there a specific permission the user needs?

6

Are this issue related?


7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.