S3-repository-path-is-not-accessible-on-master-node

Elastic Stack Elasticsearch
1

We are using 3 node cluster and we installed the repository-s3 plugin on all the 3 nodes, the ES is running in EKS cluster. We have given permission using IAM role and assigned the same using service account. When I go inside the ES pods and list the bucket, it works, so the permission is fine.

root@es-cluster-0:/usr/share/elasticsearch# bin/elasticsearch-plugin list
repository-s3

PUT _snapshot/my_repository
{
  "type": "s3",
  "settings": {
    "bucket": "servicemesh"
  }
}

{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_repository] path  is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[my_repository] path  is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object [tests-IWo1RD5zQYuf4KYEw0nGsg/master.dat] using a single upload",
      "caused_by" : {
        "type" : "amazon_service_exception",
        "reason" : "Internal Server Error (Service: null; Status Code: 500; Error Code: null; Request ID: null)"
      }
    }
  },
  "status" : 500
}

We have checked the multiple questions which are already asked by other users in the forum but nothing is worked for us.