Using s3-repository with EC2 Instance Profile and kube2iam

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
#1

I am trying to configure my Elasticsearch instance for access to a S3 bucket. I want to secure access via an AWS Role, which should be assumed automatically via kube2iam.

I am running into two problems:

  1. For kube2iam to work, it is necessary to add an annotation to the Elasticsearch Pod. I tried via the "podTemplate", but this did not work. Is there any way to do this or do you plan to add support?

  2. If I add the annotation for kube2iam manually to the Pod, I can exec into the container and verify that the AWS Role is assigned correctly. I can access to S3 bucket via aws cli. However when I try to create the repository, it fails with the following error message:

curl -H "Content-Type: application/json" -X PUT --user elastic:XXX -k https://localhost:8000/_snapshot/s3 --data '{"type":"s3", "settings": {"endpoint": "s3.eu-central-1.amazonaws.com", "bucket":"backup", "server_side_encryption": true}}'

{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[s3] path is not accessible on master node"}],"type":"repository_verification_exception","reason":"[s3] path is not accessible on master node","caused_by":{"type":"i_o_exception","reason":"Unable to upload object [tests-hOTEv96NS1yoA4mgpNgzxg/master.dat] using a single upload","caused_by":{"type":"amazon_s3_exception","reason":"Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: XXX; S3 Extended Request ID: XXX"}}},"status":500}

#2

Can you share the yaml for the pod annotation that caused issues? It should work and I verified it just now. For example, modifying the sample:

apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
...
spec:
...
    podTemplate:
      metadata:
        annotations:
          test: testvalue
      spec: {}
...

created pods with that annotation.

For your second question I am uncertain, but looking around it appears that the IAM account may be missing some necessary permissions. Here is the recommended list of permissions https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-s3-repository.html#repository-s3-permissions

#3

Thanks for clarifying that the pod annotations work. I must have made a mistake in the podTemplate specification.

The second problem is not related to IAM policies. I can verify that the elasticsearch Pod has access to the S3 bucket via the EC2 instance profile by execing into the container and running aws CLI.

However when elasticsearch tries to access the bucket via the Snapshot API it fails with aforementioned error message.

#4

I just wanted to make sure that it has all permissions, as having a subset may allow you to perform some operations in the CLI but not everything that the snapshot plugin requires. If everything looks like it's configured correctly it may be worth asking about that specifically in the Elasticsearch forums.