# Elasticsearch’s S3 Snapshot Repository: The Mystery of the Inaccessible S3 Bucket ==\> Access Denied **URL:** \ **Category:** Elasticsearch\ **Tags:** snapshot-and-restore\ **Created:** [July 10, 2024, 3:54pm UTC](https://discuss.elastic.co/t/elasticsearch-s-s3-snapshot-repository-the-mystery-of-the-inaccessible-s3-bucket-access-denied/362900 "2024-07-10T15:54:00Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![Hadj\_Hassine\_Younes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hadj_hassine_younes/32/136002_2.png) [@Hadj\_Hassine\_Younes](https://discuss.elastic.co/u/Hadj_Hassine_Younes)\ **Post date:** [July 10, 2024, 3:54pm UTC](https://discuss.elastic.co/t/elasticsearch-s-s3-snapshot-repository-the-mystery-of-the-inaccessible-s3-bucket-access-denied/362900/1 "2024-07-10T15:54:00Z")
### Elasticsearch Version 7.17.12 ### Installed Plugins repository-s3 ### Java Version JAVA\_RUNTIME\_VERSION="20.0.2+9-78 ### OS Version Linux elasticsearch-data-0 5.10.214-202.855.amzn2.x86\_64 #1 SMP Tue Apr 9 06:57:12 UTC 2024 x86\_64 x86\_64 x86\_64 GNU/Linux ### environnement I’m running Elasticsearch on an **Amazon EKS cluster** ### Problem Description I’m having a problem with Elasticsearch 7.17.12. I’m trying to set a ‘repository’ where Elasticsearch can store snapshots. But, it seems like Elasticsearch can’t access the S3 bucket I’ve set up, even though everything looks set up correctly. ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/1/111b5d326a5d131364194e6f0803277de09846cf.png) ### Steps to Reproduce I’m encountering an issue when trying to create an S3 snapshot repository in Elasticsearch 7.17.12. I’m using the S3 repository plugin of the same version. Here’s the command I’m using to create the repository: ```auto curl -X PUT "http://localhost:9200/_snapshot/elasticsearch-data-backup" -H 'Content-Type: application/json' -u 'user:password' -k -d' { "type": "s3", "settings": { "bucket": "elastic-storage-backup-dev", "endpoint": "s3.eu-west-3.amazonaws.com", "protocol": "https", "compress": true, "chunk_size": "100mb", "server_side_encryption": true } }' ``` output: ` `{"error":{"root\_cause":[{"type":"repository\_verification\_exception","reason":"[elasticsearch-data-backup] path is not accessible on master node"}],"type":"repository\_verification\_exception","reason":"[elasticsearch-data-backup] path is not accessible on master node","caused\_by":{"type":"i\_o\_exception","reason":"Unable to upload object [tests-B77q-Ol-Qz6u5jNS1YR69A/master.dat] using a single upload","caused\_by":{"type":"amazon\_s3\_exception","reason":"amazon\_s3\_exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: REZJX25X7E2TJWCZ; S3 Extended Request ID: 4e`` I’ve set up an IAM role with the necessary permissions and attached it to the Elasticsearch service. The IAM role has the following trust relationship: ```auto { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam:::oidc-provider/oidc.eks.eu-west-3.amazonaws.com/id/88D51DF3F1A6FFD0CC555D88736D4F31" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks.eu-west-3.amazonaws.com/id/88D51DF3F1A6FFD0CC555D88736D4F31:sub": [ "system:serviceaccount:default:elasticsearch-data", "system:serviceaccount:default:elasticsearch-master" ] } } } ] } ``` The S3 bucket policy is as follows: ```auto { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::role/elastic-dev" }, "Action": [ "s3:GetObject", "s3:GetBucketLocation", "s3:ListBucket", "s3:PutObject", "s3:DeleteObject", "s3:ListBucketMultipartUploads", "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts", "s3:GetObjectVersion" ], "Resource": [ "arn:aws:s3:::elastic-storage-backup-dev", "arn:aws:s3:::elastic-storage-backup-dev/*" ] } ] } ``` The Elasticsearch service and the S3 bucket are both in the eu-west-3 region. Public access is blocked for the S3 bucket. Despite this, I’m still encountering the “Access Denied” error when trying to create the S3 snapshot repository. Any help would be greatly appreciated. ### Logs (if relevant) _No response_ ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 10, 2024, 7:05pm UTC](https://discuss.elastic.co/t/elasticsearch-s-s3-snapshot-repository-the-mystery-of-the-inaccessible-s3-bucket-access-denied/362900/2 "2024-07-10T19:05:40Z")
Are you running this node on a bare EC2 instance, and if so are you attaching the IAM role to the instance so that Elasticsearch can find it via the Instance Metadata Service (IMDS)? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 10, 2024, 7:12pm UTC](https://discuss.elastic.co/t/elasticsearch-s-s3-snapshot-repository-the-mystery-of-the-inaccessible-s3-bucket-access-denied/362900/3 "2024-07-10T19:12:37Z")
My reading of the AWS Java SDK is that it fetches the credentials from the first role in the list only: > i.e. it does something like `curl http://169.254.169.254/latest/meta-data/iam/security-credentials/`, takes the role name from the first line of the response, then does `curl http://169.254.169.254/latest/meta-data/iam/security-credentials/${FIRST_ROLE_NAME}` to get the actual creds. Does this work if you do it manually? Do the credentials it returns have the right permissions? ---
**Author:** ![Hadj\_Hassine\_Younes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hadj_hassine_younes/32/136002_2.png) [@Hadj\_Hassine\_Younes](https://discuss.elastic.co/u/Hadj_Hassine_Younes)\ **Post date:** [July 11, 2024, 9:31am UTC](https://discuss.elastic.co/t/elasticsearch-s-s3-snapshot-repository-the-mystery-of-the-inaccessible-s3-bucket-access-denied/362900/4 "2024-07-11T09:31:37Z")
Hello David, thank you for your response . I’m running Elasticsearch on an Amazon EKS cluster, not directly on EC2 instances. I have set up an IAM role named `elastic-dev` with all the necessary permissions . This IAM role is associated with the service accounts `elasticsearch-data` and `elasticsearch-master` in the `default` namespace of my EKS cluster. The association is done using the `sts:AssumeRoleWithWebIdentity` action in the trust relationship of the IAM role. the service accounts elasticsearch-master and elasticsearch-data have annotation " the ARN of my iam-role : arn:aws:iam::role/elastic-dev" ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 11, 2024, 10:40am UTC](https://discuss.elastic.co/t/elasticsearch-s-s3-snapshot-repository-the-mystery-of-the-inaccessible-s3-bucket-access-denied/362900/5 "2024-07-11T10:40:22Z")
> [@Hadj\_Hassine\_Younes](#): > > I’m running Elasticsearch on an Amazon EKS cluster, not directly on EC2 instances. I see. Elasticsearch is just using the AWS Java SDK and I'm not sure if the SDK picks up IAM roles correctly in this environment. Your `AccessDenied` error suggests that it doesn't, although the few docs I've looked at suggest that this should still work. In any case I don't think we run any tests of this kind of thing in EKS so it's not really something we can help with. It's possible there's some way to adjust the SDK behaviour (e.g. with environment variables or system properties) so that it works but I'm afraid that's outside of what we might know about in terms of Elasticsearch support. One approach that'd definitely work would be to write a script which copies the role credentials from the metadata service into the Elasticsearch keystore. But again that's something outside of Elasticsearch. ---
**Author:** ![Hadj\_Hassine\_Younes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hadj_hassine_younes/32/136002_2.png) [@Hadj\_Hassine\_Younes](https://discuss.elastic.co/u/Hadj_Hassine_Younes)\ **Post date:** [July 12, 2024, 4:14pm UTC](https://discuss.elastic.co/t/elasticsearch-s-s3-snapshot-repository-the-mystery-of-the-inaccessible-s3-bucket-access-denied/362900/6 "2024-07-12T16:14:12Z")
Hello , I was going through the Elasticsearch docs and noticed that we can use service accounts for integrating with external services. > **[Service accounts | Elasticsearch Guide \[7.17\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/service-accounts.html)** So, I was wondering, can we use these service accounts with the repository-s3 plugin to authenticate to an S3 bucket? I also noticed in the Elasticsearch v7.17.12 docs, there’s no mention of using service accounts with the repository-s3 plugin. It mainly talks about IAM roles for EC2 and ECS > **[Getting Started | Elasticsearch Plugins and Integrations \[7.17\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/plugins/7.17/repository-s3-usage.html)** ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 12, 2024, 5:17pm UTC](https://discuss.elastic.co/t/elasticsearch-s-s3-snapshot-repository-the-mystery-of-the-inaccessible-s3-bucket-access-denied/362900/7 "2024-07-12T17:17:00Z")
> [@Hadj\_Hassine\_Younes](#): > > So, I was wondering, can we use these service accounts with the repository-s3 plugin to authenticate to an S3 bucket? No, those service accounts are nothing to do with S3 authentication. ---
**Author:** ![Hadj\_Hassine\_Younes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hadj_hassine_younes/32/136002_2.png) [@Hadj\_Hassine\_Younes](https://discuss.elastic.co/u/Hadj_Hassine_Younes)\ **Post date:** [July 12, 2024, 6:24pm UTC](https://discuss.elastic.co/t/elasticsearch-s-s3-snapshot-repository-the-mystery-of-the-inaccessible-s3-bucket-access-denied/362900/8 "2024-07-12T18:24:55Z")
So the repository-s3 plugin does not work with serviceaccount. What about recent versions? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [July 12, 2024, 6:49pm UTC](https://discuss.elastic.co/t/elasticsearch-s-s3-snapshot-repository-the-mystery-of-the-inaccessible-s3-bucket-access-denied/362900/9 "2024-07-12T18:49:05Z")
> [@Hadj\_Hassine\_Younes](#): > > So the repository-s3 plugin does not work with serviceaccount. > What about recent versions? The service accounts you mentioned are for internal use, they have nothing to do with external tools, you can create a service account in Elasticsearch for Fleet, Kibana and Enterprise Search and that's it. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 12, 2024, 7:09pm UTC](https://discuss.elastic.co/t/elasticsearch-s-s3-snapshot-repository-the-mystery-of-the-inaccessible-s3-bucket-access-denied/362900/10 "2024-07-12T19:09:56Z")
> [@Hadj\_Hassine\_Younes](#): > > So the repository-s3 plugin does not work with serviceaccount. > What about recent versions? Yeah what Leandro said, they're just totally unrelated. It's not a version thing. ---
**Author:** ![Hadj\_Hassine\_Younes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hadj_hassine_younes/32/136002_2.png) [@Hadj\_Hassine\_Younes](https://discuss.elastic.co/u/Hadj_Hassine_Younes)\ **Post date:** [July 12, 2024, 8:10pm UTC](https://discuss.elastic.co/t/elasticsearch-s-s3-snapshot-repository-the-mystery-of-the-inaccessible-s3-bucket-access-denied/362900/11 "2024-07-12T20:10:31Z")
Thank you guys for your response; Sorry if I asked a lot of questions, i liked the interactions. Even with eck? Because i am planning to take a snapshot from our old elasticsearch and restoring it in eck. The plugin repository-s3 is mentioned in the documentation and serviceaccount also [S3 repository | Elasticsearch Guide [8.14] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/repository-s3.html#iam-kubernetes-service-accounts). I am talking about using serviceaccount that assumes my role iam( irsa) For the old elasticsearch i can make a workarround to get the secret keys and access key from the role I created in aws and inject them in the keystore of elasticsearch. But for eck and elasticsearch 8.14.x ( which is a crd) should i do the same thing? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 13, 2024, 7:04am UTC](https://discuss.elastic.co/t/elasticsearch-s-s3-snapshot-repository-the-mystery-of-the-inaccessible-s3-bucket-access-denied/362900/12 "2024-07-13T07:04:15Z")
> [@Hadj\_Hassine\_Younes](#): > > The plugin repository-s3 is mentioned in the documentation and serviceaccount also I see, sorry I think I understand the confusion now. A "Kubernetes service account" is very much not the same thing as the "service account" mentioned in docs you linked above. I believe Kubernetes service accounts should work. ---
**Author:** ![Hadj\_Hassine\_Younes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hadj_hassine_younes/32/136002_2.png) [@Hadj\_Hassine\_Younes](https://discuss.elastic.co/u/Hadj_Hassine_Younes)\ **Post date:** [July 14, 2024, 3:47pm UTC](https://discuss.elastic.co/t/elasticsearch-s-s3-snapshot-repository-the-mystery-of-the-inaccessible-s3-bucket-access-denied/362900/13 "2024-07-14T15:47:14Z")
Hello, No i dont think that kubernets serviceaccount work with the Elasticsearch repository-s3 plugin. The plugin cannot read serviceaccount token. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 14, 2024, 8:39pm UTC](https://discuss.elastic.co/t/elasticsearch-s-s3-snapshot-repository-the-mystery-of-the-inaccessible-s3-bucket-access-denied/362900/14 "2024-07-14T20:39:32Z")
> [@Hadj\_Hassine\_Younes](#): > > No i dont think that kubernets serviceaccount work with the Elasticsearch repository-s3 plugin. The [docs you previously linked](https://www.elastic.co/guide/en/elasticsearch/reference/current/repository-s3.html#iam-kubernetes-service-accounts) disagree, and there are tests to verify that this does work. ---
**Author:** ![Hadj\_Hassine\_Younes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hadj_hassine_younes/32/136002_2.png) [@Hadj\_Hassine\_Younes](https://discuss.elastic.co/u/Hadj_Hassine_Younes)\ **Post date:** [July 14, 2024, 10:47pm UTC](https://discuss.elastic.co/t/elasticsearch-s-s3-snapshot-repository-the-mystery-of-the-inaccessible-s3-bucket-access-denied/362900/15 "2024-07-14T22:47:46Z")
Not for the 7.17.12 version of elasticsearch repository-s3 plugin. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 15, 2024, 5:28am UTC](https://discuss.elastic.co/t/elasticsearch-s-s3-snapshot-repository-the-mystery-of-the-inaccessible-s3-bucket-access-denied/362900/16 "2024-07-15T05:28:41Z")
Ah yes, it's only in 8.x indeed. The 7.x series hasn't had any new features in 2½ years, it's well worth upgrading at this point. ---
**Author:** ![Hadj\_Hassine\_Younes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hadj_hassine_younes/32/136002_2.png) [@Hadj\_Hassine\_Younes](https://discuss.elastic.co/u/Hadj_Hassine_Younes)\ **Post date:** [October 10, 2024, 8:27pm UTC](https://discuss.elastic.co/t/elasticsearch-s-s3-snapshot-repository-the-mystery-of-the-inaccessible-s3-bucket-access-denied/362900/17 "2024-10-10T20:27:20Z")
Hello @DavidTurner @leandrojmp , I created the snapshot of my old version 7.x elk stack , and I now I am trying to restore the old snapshot on the new eck-elasticsearch version 8.14.3. ### Context: - Upgraded from Elasticsearch v7.17.22 to v8.14.3 using ECK (Elastic Cloud on Kubernetes) - Configured S3 for storing snapshots - Initiated restore process from a v7.17.22 snapshot to the v8.14.3 ECK-Elasticsearch cluster ### Steps Taken: 1. Successfully restored data indices - All data indices are visible and accessible in the v8.14.3 ECK-Elasticsearch cluster 1. Attempted to restore system indices: - `.ds-.slm-history-*` - `.ds-ilm-history-*` ### Current Situation: - The system indices (`.ds-.slm-history-*` and `.ds-ilm-history-*`) were successfully restored - However, the associated data streams (`.slm-history-5` and `ilm-history-5`) are not present in the v8.14.3 cluster ### Expected Behavior: After restoring the `.ds-.slm-history-*` and `.ds-ilm-history-*` indices, we expected to see the corresponding data streams (`.slm-history-5` and `ilm-history-5`) automatically recreated in the v8.14.3 cluster. ### Question: Is this behavior expected when restoring from a 7.x snapshot to an 8.x cluster? If so, what are the recommended steps to properly recreate these system data streams while maintaining the historical data contained in the restored indices?