S3 Repository plugin works on Mac/brew but not Ubuntu/apt-get

[mooreniemi (Alex Moore-Niemi)]

I'm using ES 6.8. I have a snapshot in an S3 bucket. On Mac, using brew installed ES, I was able to successfully restore the snapshot onto my local machine. But I am not able to do the same on Ubuntu. Additionally, the error message seems rare (no or few hits on Google): Exception when listing blobs by prefix [index-]

Command and Error

I want to emphasize the below works on Mac without any problems.

untu@ip-172-31-12-105:~$ curl -X PUT "localhost:9200/_snapshot/s3_repository?pretty" -H 'Content-Type: application/json' -d'
> {
>   "type": "s3",
>   "settings": {
>     "bucket" : "my-valid-but-private-bucket-name",
>     "readonly": true
>   }
> }
> '
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[s3_repository] path  is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[s3_repository] path  is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Exception when listing blobs by prefix [index-]",
      "caused_by" : {
        "type" : "amazon_s3_exception",
        "reason" : "Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 7DE4EAD68D08E08C; S3 Extended Request ID: 0KxYlWBh+IHAiuHDkAYqqxhfEkSTKHp0rWSAswy2mTjkK5KUbqlwNW9YXJYTNBXYiFN6Oe7yRQc=)"
      }
    }
  },
  "status" : 500
}

What I've ruled out

I know that:

• The credentials are valid. (I do not want to set up other IAM permissions or policies or anything else; fundamentally, if the S3 credentials work on Mac, why don't they work on Ubuntu?)
• The object ownership is correct in the bucket (I can see the -index keys have the expected owner).
• The Ubuntu box can (independent of ES) see the snapshot files, ie. I can use aws s3 ls on the Ubuntu box and see all the snapshot files.
• The snapshot is valid. I can restore it on Mac.
• The keystore is populated (I can see the entries using list) and the plugin is installed on the single node I am running.
• I populated the keystore using the same user that runs the service in systemctl: sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch-keystore create etc.
• The bucket is in us-east-1 so should make use of default endpoint argument.

The only thing I am unsure about is I did have to change the ownership of /etc/elasticsearch which I don't see documented in the deb instructions. Without doing this, I could not create the keystore in the first place. I did not see a way around this.

[Vinayak_Sapre]

@mooreniemi
Have you reloaded secure settings? https://www.elastic.co/guide/en/elasticsearch/reference/7.8/secure-settings.html#reloadable-secure-settings

Also, are there addition error messages in elasticsearch logs?

[mooreniemi (Alex Moore-Niemi)]

I set the keys before even starting the service. But yes, I also tried reloading them.

The logs don't tell me more beyond what the overall error tells me:

[2020-07-10T22:22:16,799][WARN ][r.suppressed             ] [SQbikiG] path: /_snapshot/s3_repository, params: {pretty=, repository=s3_repository}
org.elasticsearch.repositories.RepositoryVerificationException: [s3_repository] path  is not accessible on master node
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:634) ~[elasticsearch-6.8.10.jar:6.8.10]
        at org.elasticsearch.repositories.RepositoriesService.lambda$verifyRepository$2(RepositoriesService.java:230) [elasticsearch-6.8.10.jar:6.8.10]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-6.8.10.jar:6.8.10]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.io.IOException: Exception when listing blobs by prefix [index-]
        at org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:156) ~[?:?]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:811) ~[elasticsearch-6.8.10.jar:6.8.10]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:789) ~[elasticsearch-6.8.10.jar:6.8.10]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:620) ~[elasticsearch-6.8.10.jar:6.8.10]
        ... 5 more
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 4952EF15D1D305CD; S3 Extended Request ID: nA44Jo5TWGnT1J8tn8GVLdxD4SLzO3jF1jy7pQX/UCS8S/it+CaCSgbR101L8wXVaHooMoZg6nM=)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1658) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1322) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1072) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:745) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4443) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4390) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4384) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:844) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:818) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$listBlobsByPrefix$5(S3BlobContainer.java:138) ~[?:?]

This leads me to S3BlobContainer.java#L266, but just thinking about it I can't understand why this would fail only on Ubuntu.

I did realize there is one other difference between my Mac and Ubuntu installations, it's 6.8.4 vs 6.8.10.

[mooreniemi (Alex Moore-Niemi)]

I finally figured this out. I was misunderstanding how my S3 permissions worked. I didn't actually need to set up the keystore at all. In fact, when I removed the keystore entries, it worked!

Why?

What tipped me off was this AWS documentation, specifically aws configure list. I was puzzled to see that it was picking up different credentials than I had put in my ~/.aws/credentials file. In fact, I'd set up the EC2 host with a role that had access to S3 already. So while I thought aws s3 ls was using the credentials in ~/.aws/credentials it was not because I hadn't named the profile default. This tricked me into thinking the credentials worked but something was otherwise wrong with grabbing them from the keystore.

The key was not that I was using Ubuntu but that I was on EC2 and misunderstanding which credentials were in play.