# S3 Repository fails authentication **URL:** \ **Category:** Elasticsearch\ **Created:** [April 1, 2019, 3:15pm UTC](https://discuss.elastic.co/t/s3-repository-fails-authentication/174806 "2019-04-01T15:15:59Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Pucky](https://avatars.discourse-cdn.com/v4/letter/p/e68b1a/32.png) [@Pucky](https://discuss.elastic.co/u/Pucky)\ **Post date:** [April 1, 2019, 3:16pm UTC](https://discuss.elastic.co/t/s3-repository-fails-authentication/174806/1 "2019-04-01T15:16:00Z")
Hi I'm trying to make a snapshot with my new credentials. They are failing on my cluster, but are working fine on my stand-alone node so the credentials must be ok. Is there any way that I can find out which node isn't connecting successfully? I've rebooted my cluster twice now to try and figure it out. There must be a better way to do this? ---
**Author:** ![Pucky](https://avatars.discourse-cdn.com/v4/letter/p/e68b1a/32.png) [@Pucky](https://discuss.elastic.co/u/Pucky)\ **Post date:** [April 3, 2019, 2:51pm UTC](https://discuss.elastic.co/t/s3-repository-fails-authentication/174806/2 "2019-04-03T14:51:06Z")
Any suggestions? Otherwise I have to start again with another rolling restart tomorrow. ☹ ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 3, 2019, 4:17pm UTC](https://discuss.elastic.co/t/s3-repository-fails-authentication/174806/3 "2019-04-03T16:17:55Z")
> [@Pucky](#): > > Is there any way that I can find out which node isn't connecting successfully? I think this will be recorded in the logs and/or in the response to the [repository verify API](https://www.elastic.co/guide/en/elasticsearch/reference/6.7/modules-snapshots.html#_repository_verification), i.e. `POST _snapshot//_verify`. ---
**Author:** ![Pucky](https://avatars.discourse-cdn.com/v4/letter/p/e68b1a/32.png) [@Pucky](https://discuss.elastic.co/u/Pucky)\ **Post date:** [April 4, 2019, 12:39pm UTC](https://discuss.elastic.co/t/s3-repository-fails-authentication/174806/4 "2019-04-04T12:39:50Z")
Thanks David There's no information about which node is having issues in the logs. ``` [2019-04-04T10:52:51,423][WARN][r.suppressed] path: /_snapshot/S3%20Storage/_verify, params: {repository=S3 Storage, snapshot=_verify} com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: 816BADFB91765916; S3 Extended Request ID: aC6IyvNwjvQygBVZsHCLsT39eqHKbJVCTsSB0GL35K/tcuW8sr4jIh6NMkU3TWSmB+tKZgRxrqY=) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[?:?] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4247) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4194) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4188) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:823) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:798) ~[?:?] at org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:142) ~[?:?] at org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:934) ~[elasticsearch-5.6.16.jar:5.6.16] at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:912) ~[elasticsearch-5.6.16.jar:5.6.16] at org.elasticsearch.repositories.blobstore.BlobStoreRepository.getRepositoryData(BlobStoreRepository.java:750) ~[elasticsearch-5.6.16.jar:5.6.16] at org.elasticsearch.snapshots.SnapshotsService.getRepositoryData(SnapshotsService.java:140) ~[elasticsearch-5.6.16.jar:5.6.16] at org.elasticsearch.action.admin.cluster.snapshots.get.TransportGetSnapshotsAction.masterOperation(TransportGetSnapshotsAction.java:97) [elasticsearch-5.6.16.jar:5.6.16] at org.elasticsearch.action.admin.cluster.snapshots.get.TransportGetSnapshotsAction.masterOperation(TransportGetSnapshotsAction.java:55) [elasticsearch-5.6.16.jar:5.6.16] at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:87) [elasticsearch-5.6.16.jar:5.6.16] at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:166) [elasticsearch-5.6.16.jar:5.6.16] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:675) [elasticsearch-5.6.16.jar:5.6.16] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.16.jar:5.6.16] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171] ``` ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 4, 2019, 12:41pm UTC](https://discuss.elastic.co/t/s3-repository-fails-authentication/174806/5 "2019-04-04T12:41:47Z")
I think this message indicates that the master node itself cannot access your repository on S3. The message strongly suggests a configuration issue: > [@Pucky](#): > > `The AWS Access Key Id you provided does not exist in our records` ---
**Author:** ![Pucky](https://avatars.discourse-cdn.com/v4/letter/p/e68b1a/32.png) [@Pucky](https://discuss.elastic.co/u/Pucky)\ **Post date:** [April 8, 2019, 9:28am UTC](https://discuss.elastic.co/t/s3-repository-fails-authentication/174806/6 "2019-04-08T09:28:20Z")
I have gone through each server again and updated the credentials, restarted them all and still no luck so there must be something else wrong. I've tested my credentials with a separate service and they work fine. All my servers have unfettered access to the internet. If they are blocked at all it would be at Amazon. I see no evidence of that. The most likely thing is that Elasticsearch is either not able to read the keystores or is reading a different keystore to what the keytool is altering. I can't see another keystore on the servers, unless the naming scheme changed at some point, these servers have been around since 2.x. Any thoughts? ---
**Author:** ![Pucky](https://avatars.discourse-cdn.com/v4/letter/p/e68b1a/32.png) [@Pucky](https://discuss.elastic.co/u/Pucky)\ **Post date:** [April 8, 2019, 10:07am UTC](https://discuss.elastic.co/t/s3-repository-fails-authentication/174806/7 "2019-04-08T10:07:21Z")
I entered the credentials in cerebro and it updated the setting using the API. Now it works. I guess it wasn't looking at the keystore at all. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 6, 2019, 10:07am UTC](https://discuss.elastic.co/t/s3-repository-fails-authentication/174806/8 "2019-05-06T10:07:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.