# S3 backup configuration error **URL:** \ **Category:** Elasticsearch\ **Tags:** snapshot-and-restore\ **Created:** [February 1, 2021, 8:01am UTC](https://discuss.elastic.co/t/s3-backup-configuration-error/262795 "2021-02-01T08:01:39Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Lior\_Yakobov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lior_yakobov/32/47279_2.png) [@Lior\_Yakobov](https://discuss.elastic.co/u/Lior_Yakobov)\ **Post date:** [February 1, 2021, 8:01am UTC](https://discuss.elastic.co/t/s3-backup-configuration-error/262795/1 "2021-02-01T08:01:39Z")
Hello, I'm trying to configure AWS S3 repository for backups but facing access issues to the S3 bucket. I did the following steps: 1. Installed repository-s3 plugin on all cluster nodes 2. restarted all nodes after the plugin installation 3. I'm using elasticsearch-keystore to configure the credentials for S3, I have set up both **s3.client.default.access\_key** and **s3.client.default.secret\_key**. 4. Configuring the repository through Kibana and getting this error when I'm trying to test the repository: ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/9/198ad24543c66024d05e2b39bbc82e879e67433c.png) Seems that it tries to get the credentials from attached IAM role rather than using the credentials in the keystore. How can I make it consider the keystore configuration? Thanks, Lior ---
**Author:** ![afharo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/afharo/32/75202_2.png) [@afharo](https://discuss.elastic.co/u/afharo)\ **Post date:** [February 1, 2021, 3:34pm UTC](https://discuss.elastic.co/t/s3-backup-configuration-error/262795/2 "2021-02-01T15:34:48Z")
Hi @Lior_Yakobov, Can you confirm that you've set up the S3 credentials in all ES nodes? Thank you! ---
**Author:** ![Lior\_Yakobov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lior_yakobov/32/47279_2.png) [@Lior\_Yakobov](https://discuss.elastic.co/u/Lior_Yakobov)\ **Post date:** [February 1, 2021, 3:43pm UTC](https://discuss.elastic.co/t/s3-backup-configuration-error/262795/3 "2021-02-01T15:43:14Z")
Hey @afharo, Yes I have created the keystore pairs on all cluster nodes. Does the creation of keystore pairs also requires restart to Elasticsearch process in order for Elasticsearch to recognize it? Thanks, Lior ---
**Author:** ![afharo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/afharo/32/75202_2.png) [@afharo](https://discuss.elastic.co/u/afharo)\ **Post date:** [February 1, 2021, 4:00pm UTC](https://discuss.elastic.co/t/s3-backup-configuration-error/262795/4 "2021-02-01T16:00:09Z")
There is no need to restart Elasticsearch, although you might need to call the `/reload_secure_settings` API to get all the nodes to re-read them: [Secure settings | Elasticsearch Reference [7.10] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/secure-settings.html#reloadable-secure-settings) ---
**Author:** ![Lior\_Yakobov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lior_yakobov/32/47279_2.png) [@Lior\_Yakobov](https://discuss.elastic.co/u/Lior_Yakobov)\ **Post date:** [February 2, 2021, 7:25am UTC](https://discuss.elastic.co/t/s3-backup-configuration-error/262795/5 "2021-02-02T07:25:42Z")
Hey @afharo, Thanks for the heads-up, I probably missed it. So after running the reload\_secure\_settings request I do see that test files were created in the S3 bucket, but still Kibana complains about something else now: ``` { "error": { "root_cause": [ { "type": "repository_verification_exception", "reason": "[bucket-elasticsearch-repo] [[DmPiHBoGSxWOxcAUnSEHKw, 'RemoteTransportException[[aws-elkdb22][10.128.115.52:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[bucket-elasticsearch-repo] missing];']]" } ], "type": "repository_verification_exception", "reason": "[bucket-elasticsearch-repo] [[DmPiHBoGSxWOxcAUnSEHKw, 'RemoteTransportException[[aws-elkdb22][10.128.115.52:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[bucket-elasticsearch-repo] missing];']]" }, "status": 500 } ``` Is there something else I'm missing here? Thanks again, Lior ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [February 2, 2021, 7:54am UTC](https://discuss.elastic.co/t/s3-backup-configuration-error/262795/6 "2021-02-02T07:54:01Z")
What version are you running? Do you have any voting-only nodes in the cluster? ---
**Author:** ![Lior\_Yakobov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lior_yakobov/32/47279_2.png) [@Lior\_Yakobov](https://discuss.elastic.co/u/Lior_Yakobov)\ **Post date:** [February 2, 2021, 7:59am UTC](https://discuss.elastic.co/t/s3-backup-configuration-error/262795/7 "2021-02-02T07:59:02Z")
Hey @DavidTurner, our cluster is version 7.9.3, and I believe that by voting-only nodes you mean the Kibana nodes, as they configured this way: ``` node.data: false node.master: false node.ingest: true node.ml: false ``` If that's what you meant, so yes we have 2 of these kind of nodes. Thanks, Lior ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [February 2, 2021, 8:01am UTC](https://discuss.elastic.co/t/s3-backup-configuration-error/262795/8 "2021-02-02T08:01:48Z")
No I meant nodes with `node.voting_only: true`. Any of them? ---
**Author:** ![Lior\_Yakobov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lior_yakobov/32/47279_2.png) [@Lior\_Yakobov](https://discuss.elastic.co/u/Lior_Yakobov)\ **Post date:** [February 2, 2021, 8:27am UTC](https://discuss.elastic.co/t/s3-backup-configuration-error/262795/9 "2021-02-02T08:27:25Z")
Hey @DavidTurner, actually I don't have nodes of this type in the cluster. Lior ---
**Author:** ![Lior\_Yakobov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lior_yakobov/32/47279_2.png) [@Lior\_Yakobov](https://discuss.elastic.co/u/Lior_Yakobov)\ **Post date:** [February 2, 2021, 1:49pm UTC](https://discuss.elastic.co/t/s3-backup-configuration-error/262795/10 "2021-02-02T13:49:03Z")
Hey @DavidTurner , @afharo despite the message I received from testing the repository, seems that I managed to perform a successful backup. I will try to complete a recovery as well and if all goes well then we're good to go. Thanks, Lior ---
**Author:** ![Lior\_Yakobov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lior_yakobov/32/47279_2.png) [@Lior\_Yakobov](https://discuss.elastic.co/u/Lior_Yakobov)\ **Post date:** [February 3, 2021, 1:22pm UTC](https://discuss.elastic.co/t/s3-backup-configuration-error/262795/11 "2021-02-03T13:22:11Z")
Hey @DavidTurner, @afharo, So I managed to perform both snapshot and restore, although when I'm pressing the repository verification I still get this error message: ``` { "error": { "root_cause": [ { "type": "repository_verification_exception", "reason": "[bucket-elasticsearch-repo] [[DmPiHBoGSxWOxcAUnSEHKw, 'RemoteTransportException[[aws-elkdb22][10.128.115.52:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[bucket-elasticsearch-repo] missing];']]" } ], "type": "repository_verification_exception", "reason": "[bucket-elasticsearch-repo] [[DmPiHBoGSxWOxcAUnSEHKw, 'RemoteTransportException[[aws-elkdb22][10.128.115.52:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[bucket-elasticsearch-repo] missing];']]" }, "status": 500 } ``` Is there any reason for this error message to appear, even though snapshots are working? Thanks, Lior ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [February 3, 2021, 1:48pm UTC](https://discuss.elastic.co/t/s3-backup-configuration-error/262795/12 "2021-02-03T13:48:35Z")
> [@Lior\_Yakobov](#): > > `aws-elkdb22` There's some kind of discrepancy with the config of `aws-elkdb22` vs how it appears in the cluster state. If you restart that one node does the problem go away? ---
**Author:** ![Lior\_Yakobov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lior_yakobov/32/47279_2.png) [@Lior\_Yakobov](https://discuss.elastic.co/u/Lior_Yakobov)\ **Post date:** [February 3, 2021, 2:42pm UTC](https://discuss.elastic.co/t/s3-backup-configuration-error/262795/13 "2021-02-03T14:42:39Z")
@DavidTurner, Thank you for the help, by restarting this node I noticed in the log file that it refuses to start since **repository-s3** plugin was missing. I guess that somehow I missed the plugin installation on this specific node, but now everything looks just fine. Thanks again and best regards, Lior ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 3, 2021, 2:43pm UTC](https://discuss.elastic.co/t/s3-backup-configuration-error/262795/14 "2021-03-03T14:43:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.