Permission error when registering S3 bucket as snapshot repository

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hello everyone! :waving_hand:
I hope you’re all doing well.

I’m currently trying to register an S3 bucket as a snapshot repository for my Elasticsearch backups, but I’m running into a permission error.

Below are my configurations, the IAM policy, the error message, and the documentation I used as a reference.
I’m using AWS IAM IRSA for authentication.

Master Node Configuration

--
clusterName: "prod-es"

nodeGroup: "master"

roles:
  - master
replicas: 1

minimumMasterNodes: 1

esMajorVersion: "7.17.15"

createCert: true
secret:
  enabled: false

extraInitContainers:
  - name: copy-password
    image: busybox:1.35
    command: ['sh', '-c', 'cp /mnt/elastic-password/elastic-password /shared/elastic-password && chown 1000:1000 /shared/elastic-password && chmod 600 /shared/elastic-password']
    volumeMounts:
      - name: elastic-password-vol
        mountPath: "/mnt/elastic-password"
        readOnly: true
      - name: shared-data
        mountPath: "/shared"
    securityContext:
      runAsUser: 0
  - name: install-elasticsearch-plugins
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.15
    command: 
      - sh
      - -c
      - |
        yes | bin/elasticsearch-plugin install repository-s3 --batch
    volumeMounts:
      - name: plugins
        mountPath: "/usr/share/elasticsearch/plugins"

protocol: http
extraEnvs:
  - name: ELASTIC_PASSWORD_FILE
    value: /shared/elastic-password
  - name: AWS_ROLE_ARN
    value: "arn:aws:iam::xxxxxxxxxx:role/pod-role"
  - name: ingest.geoip.downloader.enabled
    value: "false"
  - name: http.cors.enabled
    value: "true"
  - name: http.cors.allow-origin
    value: "\"*\""
  - name: xpack.monitoring.collection.enabled
    value: "true"
  - name: bootstrap.memory_lock
    value: "true"
  - name: xpack.security.http.ssl.enabled 
    value: "false" 
  - name: xpack.security.transport.ssl.verification_mode 
    value: "none" 

imageTag: "7.17.15"

esJavaOpts: "-Xmx670m -Xms670m"
readinessProbe:
  exec:
    command:
      - bash
      - -c
      - |
        # Verificar se o Elasticsearch está respondendo e autenticado
        curl -s -u 'elastic:$(cat /shared/elastic-password)' http://localhost:9200/_cluster/health?wait_for_status=green&timeout=1s
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3


resources:
  requests:
    cpu: "815m"
    memory: "1300Mi"
  limits:
    cpu: "815m"
    memory: "1300Mi"

volumeClaimTemplate:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: "30Gi"
  storageClassName: gp3

nodeAffinity: 
  requiredDuringSchedulingIgnoredDuringExecution:
    nodeSelectorTerms:
      - matchExpressions:
        - key: node.kubernetes.io/instance-type
          operator: In
          values:
            - c7g.large

rbac:
  create: false
  serviceAccountAnnotations: {}
  serviceAccountName: "pod-service-account"
  automountToken: true

extraVolumes:
  - name: elastic-password-vol
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "elastic-password"
  - name: shared-data
    emptyDir: {}
  - name: plugins
    emptyDir: {}
  - name: aws-iam-token
    projected:
      sources:
        - serviceAccountToken:
            audience: sts.amazonaws.com
            expirationSeconds: 86400
            path: aws-web-identity-token-file

extraVolumeMounts:
  - name: elastic-password-vol
    mountPath: "/mnt/elastic-password"
    readOnly: true
  - name: shared-data
    mountPath: "/shared"
  - name: plugins
    mountPath: "/usr/share/elasticsearch/plugins"
    readOnly: false
  - name: aws-iam-token
    mountPath: "/usr/share/elasticsearch/config/repository-s3"

podSecurityContext:
  runAsUser: 1000

IAM Policy



{
        "Sid": "VisualEditor0",
        "Effect": "Allow",
        "Action": [
          "s3:ListBucketMultipartUploads",
          "s3:ListBucketVersions",
          "s3:ListBucket",
          "s3:GetBucketLocation"
        ],
        "Resource": "arn:aws:s3:::pd-elasticsearch-devops-prod-auth"
      },
      {
        "Sid": "VisualEditor1",
        "Effect": "Allow",
        "Action": [
          "s3:PutObject",
          "s3:GetObject",
          "s3:AbortMultipartUpload",
          "s3:DeleteObject",
          "s3:ListMultipartUploadParts"
        ],
        "Resource": "arn:aws:s3:::pd-elasticsearch-devops-prod-auth/*"
      },
      {
        "Effect": "Allow",
        "Action": [
          "ssm:*",
          "secretsmanager:GetSecretValue",
          "ecr:GetAuthorizationToken",
          "ecr:BatchCheckLayerAvailability",
          "ecr:GetDownloadUrlForLayer",
          "ecr:GetRepositoryPolicy",
          "ecr:DescribeRepositories",
          "ecr:ListImages",
          "ecr:BatchGetImage"
        ],
        "Resource": "*"
      }

Error Message

{
    "error": {
        "root_cause": [
            {
                "type": "repository_verification_exception",
                "reason": "[els-backup] path [els-backup] is not accessible on master node"
            }
        ],
        "type": "repository_verification_exception",
        "reason": "[els-backup] path [els-backup] is not accessible on master node",
        "caused_by": {
            "type": "i_o_exception",
            "reason": "Unable to upload object [els-backup/tests-quT3nbyKTyyWvKbK7K_bkw/master.dat] using a single upload",
            "caused_by": {
                "type": "amazon_service_exception",
                "reason": "amazon_service_exception: Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null)"
            }
        }
    },
    "status": 500
}

Documentation Used
Elastic Docs – S3 Repository Configuration

Does anyone know what might be causing this permission or authentication issue with IRSA?
Any guidance or suggestions would be greatly appreciated. :folded_hands: