Elasticsearch’s S3 Snapshot Repository: The Mystery of the Inaccessible S3 Bucket ==> Access Denied

Elastic Stack Elasticsearch
1

Elasticsearch Version

7.17.12

Installed Plugins

repository-s3

Java Version

JAVA_RUNTIME_VERSION="20.0.2+9-78

OS Version

Linux elasticsearch-data-0 5.10.214-202.855.amzn2.x86_64 #1 SMP Tue Apr 9 06:57:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

environnement

I’m running Elasticsearch on an Amazon EKS cluster

Problem Description

I’m having a problem with Elasticsearch 7.17.12. I’m trying to set a ‘repository’ where Elasticsearch can store snapshots.
But, it seems like Elasticsearch can’t access the S3 bucket I’ve set up, even though everything looks set up correctly.

image
image609×785 64.1 KB

Steps to Reproduce

I’m encountering an issue when trying to create an S3 snapshot repository in Elasticsearch 7.17.12. I’m using the S3 repository plugin of the same version.

Here’s the command I’m using to create the repository:

curl -X PUT "http://localhost:9200/_snapshot/elasticsearch-data-backup" -H 'Content-Type: application/json' -u 'user:password' -k  -d' 
{
  "type": "s3",
  "settings": {
    "bucket": "elastic-storage-backup-dev",
    "endpoint": "s3.eu-west-3.amazonaws.com",
    "protocol": "https",
    "compress": true,
    "chunk_size": "100mb",
    "server_side_encryption": true 
    }
}'

output:
{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[elasticsearch-data-backup] path is not accessible on master node"}],"type":"repository_verification_exception","reason":"[elasticsearch-data-backup] path is not accessible on master node","caused_by":{"type":"i_o_exception","reason":"Unable to upload object [tests-B77q-Ol-Qz6u5jNS1YR69A/master.dat] using a single upload","caused_by":{"type":"amazon_s3_exception","reason":"amazon_s3_exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: REZJX25X7E2TJWCZ; S3 Extended Request ID: 4e``

I’ve set up an IAM role with the necessary permissions and attached it to the Elasticsearch service. The IAM role has the following trust relationship:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::<account-id>:oidc-provider/oidc.eks.eu-west-3.amazonaws.com/id/88D51DF3F1A6FFD0CC555D88736D4F31"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.eu-west-3.amazonaws.com/id/88D51DF3F1A6FFD0CC555D88736D4F31:sub": [
                        "system:serviceaccount:default:elasticsearch-data",
                        "system:serviceaccount:default:elasticsearch-master"
                    ]
                }
            }
        }
    ]
}

The S3 bucket policy is as follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<account-id>role/elastic-dev"
            },
            "Action": [
                "s3:GetObject",
                "s3:GetBucketLocation",
                "s3:ListBucket",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucketMultipartUploads",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts",
                "s3:GetObjectVersion"
            ],
            "Resource": [
                "arn:aws:s3:::elastic-storage-backup-dev",
                "arn:aws:s3:::elastic-storage-backup-dev/*"
            ]
        }
    ]
}

The Elasticsearch service and the S3 bucket are both in the eu-west-3 region. Public access is blocked for the S3 bucket.

Despite this, I’m still encountering the “Access Denied” error when trying to create the S3 snapshot repository. Any help would be greatly appreciated.

Logs (if relevant)

No response

2

Are you running this node on a bare EC2 instance, and if so are you attaching the IAM role to the instance so that Elasticsearch can find it via the Instance Metadata Service (IMDS)?

3

My reading of the AWS Java SDK is that it fetches the credentials from the first role in the list only:

i.e. it does something like curl http://169.254.169.254/latest/meta-data/iam/security-credentials/, takes the role name from the first line of the response, then does curl http://169.254.169.254/latest/meta-data/iam/security-credentials/${FIRST_ROLE_NAME} to get the actual creds. Does this work if you do it manually? Do the credentials it returns have the right permissions?

4

Hello David,
thank you for your response .
I’m running Elasticsearch on an Amazon EKS cluster, not directly on EC2 instances.
I have set up an IAM role named elastic-dev with all the necessary permissions .
This IAM role is associated with the service accounts elasticsearch-data and elasticsearch-master in the default namespace of my EKS cluster. The association is done using the sts:AssumeRoleWithWebIdentity action in the trust relationship of the IAM role.
the service accounts elasticsearch-master and elasticsearch-data have annotation " the ARN of my iam-role : arn:aws:iam::role/elastic-dev"

5

I see. Elasticsearch is just using the AWS Java SDK and I'm not sure if the SDK picks up IAM roles correctly in this environment. Your AccessDenied error suggests that it doesn't, although the few docs I've looked at suggest that this should still work. In any case I don't think we run any tests of this kind of thing in EKS so it's not really something we can help with.

It's possible there's some way to adjust the SDK behaviour (e.g. with environment variables or system properties) so that it works but I'm afraid that's outside of what we might know about in terms of Elasticsearch support.

One approach that'd definitely work would be to write a script which copies the role credentials from the metadata service into the Elasticsearch keystore. But again that's something outside of Elasticsearch.

6

Hello ,

I was going through the Elasticsearch docs and noticed that we can use service accounts for integrating with external services.

So, I was wondering, can we use these service accounts with the repository-s3 plugin to authenticate to an S3 bucket?

I also noticed in the Elasticsearch v7.17.12 docs, there’s no mention of using service accounts with the repository-s3 plugin. It mainly talks about IAM roles for EC2 and ECS

7

No, those service accounts are nothing to do with S3 authentication.

8

So the repository-s3 plugin does not work with serviceaccount.
What about recent versions?

9

The service accounts you mentioned are for internal use, they have nothing to do with external tools, you can create a service account in Elasticsearch for Fleet, Kibana and Enterprise Search and that's it.

10

Yeah what Leandro said, they're just totally unrelated. It's not a version thing.

11

Thank you guys for your response;

Sorry if I asked a lot of questions, i liked the interactions.

Even with eck? Because i am planning to take a snapshot from our old elasticsearch and restoring it in eck.
The plugin repository-s3 is mentioned in the documentation and serviceaccount also
S3 repository | Elasticsearch Guide [8.14] | Elastic.
I am talking about using serviceaccount that assumes my role iam( irsa)
For the old elasticsearch i can make a workarround to get the secret keys and access key from the role I created in aws and inject them in the keystore of elasticsearch.
But for eck and elasticsearch 8.14.x ( which is a crd) should i do the same thing?

12

I see, sorry I think I understand the confusion now. A "Kubernetes service account" is very much not the same thing as the "service account" mentioned in docs you linked above. I believe Kubernetes service accounts should work.

13

Hello,
No i dont think that kubernets serviceaccount work with the Elasticsearch repository-s3 plugin.
The plugin cannot read serviceaccount token.

14

The docs you previously linked disagree, and there are tests to verify that this does work.

15

Not for the 7.17.12 version of elasticsearch repository-s3 plugin.

16

Ah yes, it's only in 8.x indeed. The 7.x series hasn't had any new features in 2½ years, it's well worth upgrading at this point.