Elasticsearch’s S3 Snapshot Repository: The Mystery of the Inaccessible S3 Bucket ==> Access Denied

Hadj_Hassine_Younes · July 10, 2024, 3:54pm

Elasticsearch Version

7.17.12

Installed Plugins

repository-s3

Java Version

JAVA_RUNTIME_VERSION="20.0.2+9-78

OS Version

Linux elasticsearch-data-0 5.10.214-202.855.amzn2.x86_64 #1 SMP Tue Apr 9 06:57:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

environnement

I’m running Elasticsearch on an Amazon EKS cluster

Problem Description

I’m having a problem with Elasticsearch 7.17.12. I’m trying to set a ‘repository’ where Elasticsearch can store snapshots.
But, it seems like Elasticsearch can’t access the S3 bucket I’ve set up, even though everything looks set up correctly.

Steps to Reproduce

I’m encountering an issue when trying to create an S3 snapshot repository in Elasticsearch 7.17.12. I’m using the S3 repository plugin of the same version.

Here’s the command I’m using to create the repository:

curl -X PUT "http://localhost:9200/_snapshot/elasticsearch-data-backup" -H 'Content-Type: application/json' -u 'user:password' -k  -d' 
{
  "type": "s3",
  "settings": {
    "bucket": "elastic-storage-backup-dev",
    "endpoint": "s3.eu-west-3.amazonaws.com",
    "protocol": "https",
    "compress": true,
    "chunk_size": "100mb",
    "server_side_encryption": true 
    }
}'

output:
{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[elasticsearch-data-backup] path is not accessible on master node"}],"type":"repository_verification_exception","reason":"[elasticsearch-data-backup] path is not accessible on master node","caused_by":{"type":"i_o_exception","reason":"Unable to upload object [tests-B77q-Ol-Qz6u5jNS1YR69A/master.dat] using a single upload","caused_by":{"type":"amazon_s3_exception","reason":"amazon_s3_exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: REZJX25X7E2TJWCZ; S3 Extended Request ID: 4e``

I’ve set up an IAM role with the necessary permissions and attached it to the Elasticsearch service. The IAM role has the following trust relationship:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::<account-id>:oidc-provider/oidc.eks.eu-west-3.amazonaws.com/id/88D51DF3F1A6FFD0CC555D88736D4F31"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.eu-west-3.amazonaws.com/id/88D51DF3F1A6FFD0CC555D88736D4F31:sub": [
                        "system:serviceaccount:default:elasticsearch-data",
                        "system:serviceaccount:default:elasticsearch-master"
                    ]
                }
            }
        }
    ]
}

The S3 bucket policy is as follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<account-id>role/elastic-dev"
            },
            "Action": [
                "s3:GetObject",
                "s3:GetBucketLocation",
                "s3:ListBucket",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucketMultipartUploads",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts",
                "s3:GetObjectVersion"
            ],
            "Resource": [
                "arn:aws:s3:::elastic-storage-backup-dev",
                "arn:aws:s3:::elastic-storage-backup-dev/*"
            ]
        }
    ]
}

The Elasticsearch service and the S3 bucket are both in the eu-west-3 region. Public access is blocked for the S3 bucket.

Despite this, I’m still encountering the “Access Denied” error when trying to create the S3 snapshot repository. Any help would be greatly appreciated.

Logs (if relevant)

No response

DavidTurner · July 10, 2024, 7:05pm

Are you running this node on a bare EC2 instance, and if so are you attaching the IAM role to the instance so that Elasticsearch can find it via the Instance Metadata Service (IMDS)?

DavidTurner · July 10, 2024, 7:12pm

My reading of the AWS Java SDK is that it fetches the credentials from the first role in the list only:

github.com

aws/aws-sdk-java/blob/98353304a6e0e10bf4c11f121bd04a9fac84e42b/aws-java-sdk-core/src/main/java/com/amazonaws/auth/InstanceMetadataServiceCredentialsFetcher.java#L48-L66


      
          protected String getCredentialsResponse() {
              URI credentialsEndpoint = getCredentialsEndpoint();
              return resourceFetcher.readResource(credentialsEndpoint, this);
          }
          
          @Override
          public String toString() {
              return "InstanceMetadataServiceCredentialsFetcher";
          }
          
          private URI getCredentialsEndpoint() {
              String host = EC2MetadataUtils.getHostAddressForEC2MetadataService();
          
              String securityCredentialsList = resourceFetcher.readResource(URI.create(host + EC2MetadataUtils.SECURITY_CREDENTIALS_RESOURCE), this);
          
              String[] securityCredentials = securityCredentialsList.trim().split("\n");
              if (securityCredentials.length == 0) {
                  throw new SdkClientException("Unable to load credentials path");
              }

i.e. it does something like curl http://169.254.169.254/latest/meta-data/iam/security-credentials/, takes the role name from the first line of the response, then does curl http://169.254.169.254/latest/meta-data/iam/security-credentials/${FIRST_ROLE_NAME} to get the actual creds. Does this work if you do it manually? Do the credentials it returns have the right permissions?

Hadj_Hassine_Younes · July 11, 2024, 9:31am

Hello David,
thank you for your response .
I’m running Elasticsearch on an Amazon EKS cluster, not directly on EC2 instances.
I have set up an IAM role named elastic-dev with all the necessary permissions .
This IAM role is associated with the service accounts elasticsearch-data and elasticsearch-master in the default namespace of my EKS cluster. The association is done using the sts:AssumeRoleWithWebIdentity action in the trust relationship of the IAM role.
the service accounts elasticsearch-master and elasticsearch-data have annotation " the ARN of my iam-role : arn:aws:iam::role/elastic-dev"

DavidTurner · July 11, 2024, 10:40am

I see. Elasticsearch is just using the AWS Java SDK and I'm not sure if the SDK picks up IAM roles correctly in this environment. Your AccessDenied error suggests that it doesn't, although the few docs I've looked at suggest that this should still work. In any case I don't think we run any tests of this kind of thing in EKS so it's not really something we can help with.

It's possible there's some way to adjust the SDK behaviour (e.g. with environment variables or system properties) so that it works but I'm afraid that's outside of what we might know about in terms of Elasticsearch support.

One approach that'd definitely work would be to write a script which copies the role credentials from the metadata service into the Elasticsearch keystore. But again that's something outside of Elasticsearch.

Hadj_Hassine_Younes · July 12, 2024, 4:14pm

Hello ,

I was going through the Elasticsearch docs and noticed that we can use service accounts for integrating with external services.

So, I was wondering, can we use these service accounts with the repository-s3 plugin to authenticate to an S3 bucket?

I also noticed in the Elasticsearch v7.17.12 docs, there’s no mention of using service accounts with the repository-s3 plugin. It mainly talks about IAM roles for EC2 and ECS

DavidTurner · July 12, 2024, 5:17pm

No, those service accounts are nothing to do with S3 authentication.

Hadj_Hassine_Younes · July 12, 2024, 6:24pm

So the repository-s3 plugin does not work with serviceaccount.
What about recent versions?

leandrojmp · July 12, 2024, 6:49pm

The service accounts you mentioned are for internal use, they have nothing to do with external tools, you can create a service account in Elasticsearch for Fleet, Kibana and Enterprise Search and that's it.

DavidTurner · July 12, 2024, 7:09pm

Yeah what Leandro said, they're just totally unrelated. It's not a version thing.

Hadj_Hassine_Younes · July 12, 2024, 8:10pm

Thank you guys for your response;

Sorry if I asked a lot of questions, i liked the interactions.

Even with eck? Because i am planning to take a snapshot from our old elasticsearch and restoring it in eck.
The plugin repository-s3 is mentioned in the documentation and serviceaccount also
S3 repository | Elasticsearch Guide [8.14] | Elastic.
I am talking about using serviceaccount that assumes my role iam( irsa)
For the old elasticsearch i can make a workarround to get the secret keys and access key from the role I created in aws and inject them in the keystore of elasticsearch.
But for eck and elasticsearch 8.14.x ( which is a crd) should i do the same thing?

DavidTurner · July 13, 2024, 7:04am

I see, sorry I think I understand the confusion now. A "Kubernetes service account" is very much not the same thing as the "service account" mentioned in docs you linked above. I believe Kubernetes service accounts should work.

Hadj_Hassine_Younes · July 14, 2024, 3:47pm

Hello,
No i dont think that kubernets serviceaccount work with the Elasticsearch repository-s3 plugin.
The plugin cannot read serviceaccount token.

DavidTurner · July 14, 2024, 8:39pm

The docs you previously linked disagree, and there are tests to verify that this does work.

Hadj_Hassine_Younes · July 14, 2024, 10:47pm

Not for the 7.17.12 version of elasticsearch repository-s3 plugin.

DavidTurner · July 15, 2024, 5:28am

Ah yes, it's only in 8.x indeed. The 7.x series hasn't had any new features in 2½ years, it's well worth upgrading at this point.

Hadj_Hassine_Younes · October 10, 2024, 8:27pm

Hello @DavidTurner @leandrojmp ,

I created the snapshot of my old version 7.x elk stack , and I now I am trying to restore the old snapshot on the new eck-elasticsearch version 8.14.3.

Context:

Upgraded from Elasticsearch v7.17.22 to v8.14.3 using ECK (Elastic Cloud on Kubernetes)
Configured S3 for storing snapshots
Initiated restore process from a v7.17.22 snapshot to the v8.14.3 ECK-Elasticsearch cluster

Steps Taken:

Successfully restored data indices

All data indices are visible and accessible in the v8.14.3 ECK-Elasticsearch cluster

Attempted to restore system indices:

.ds-.slm-history-*
.ds-ilm-history-*

Current Situation:

The system indices (.ds-.slm-history-* and .ds-ilm-history-*) were successfully restored
However, the associated data streams (.slm-history-5 and ilm-history-5) are not present in the v8.14.3 cluster

Expected Behavior:

After restoring the .ds-.slm-history-* and .ds-ilm-history-* indices, we expected to see the corresponding data streams (.slm-history-5 and ilm-history-5) automatically recreated in the v8.14.3 cluster.

Question:

Is this behavior expected when restoring from a 7.x snapshot to an 8.x cluster? If so, what are the recommended steps to properly recreate these system data streams while maintaining the historical data contained in the restored indices?

Topic		Replies	Views
S3 Access Denied error while verifying repository for snapshot and restore Elasticsearch snapshot-and-restore	1	528	May 2, 2023
Elasticsearch s3 snapshot Elasticsearch elastic-stack-searchable-snapshots	4	2677	July 14, 2021
Elasticsearch snapshot/restore to s3 Elasticsearch snapshot-and-restore	31	3572	June 14, 2023
How can I restore a snapshot from S3 to a different cluster? Elasticsearch snapshot-and-restore	10	34	October 4, 2024
Cannot list snapshots in S3 repository (previously working fine) Elasticsearch	5	2913	November 16, 2017