S3 Repository fails authentication

Elastic Stack Elasticsearch
1

Hi

I'm trying to make a snapshot with my new credentials. They are failing on my cluster, but are working fine on my stand-alone node so the credentials must be ok. Is there any way that I can find out which node isn't connecting successfully? I've rebooted my cluster twice now to try and figure it out. There must be a better way to do this?

2

Any suggestions? Otherwise I have to start again with another rolling restart tomorrow. :frowning:

3

I think this will be recorded in the logs and/or in the response to the repository verify API, i.e. POST _snapshot/<REPOSITORY>/_verify.

4

Thanks David

There's no information about which node is having issues in the logs.

[2019-04-04T10:52:51,423][WARN ][r.suppressed             ] path: /_snapshot/S3%20Storage/_verify, params: {repository=S3 Storage, snapshot=_verify}
com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: 816BADFB91765916; S3 Extended Request ID: aC6IyvNwjvQygBVZsHCLsT39eqHKbJVCTsSB0GL35K/tcuW8sr4jIh6NMkU3TWSmB+tKZgRxrqY=)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4247) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4194) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4188) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:823) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:798) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:142) ~[?:?]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:934) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:912) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.getRepositoryData(BlobStoreRepository.java:750) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.snapshots.SnapshotsService.getRepositoryData(SnapshotsService.java:140) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.action.admin.cluster.snapshots.get.TransportGetSnapshotsAction.masterOperation(TransportGetSnapshotsAction.java:97) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.action.admin.cluster.snapshots.get.TransportGetSnapshotsAction.masterOperation(TransportGetSnapshotsAction.java:55) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:87) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:166) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:675) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.16.jar:5.6.16]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
5

I think this message indicates that the master node itself cannot access your repository on S3. The message strongly suggests a configuration issue:

6

I have gone through each server again and updated the credentials, restarted them all and still no luck so there must be something else wrong.

I've tested my credentials with a separate service and they work fine.
All my servers have unfettered access to the internet. If they are blocked at all it would be at Amazon. I see no evidence of that.
The most likely thing is that Elasticsearch is either not able to read the keystores or is reading a different keystore to what the keytool is altering. I can't see another keystore on the servers, unless the naming scheme changed at some point, these servers have been around since 2.x.

Any thoughts?

7

I entered the credentials in cerebro and it updated the setting using the API. Now it works. I guess it wasn't looking at the keystore at all.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.