Amazon_s3_exception: Access Denied Service: Amazon S3; Status Code: 403;

Elastic Stack Elasticsearch
1

I have a 3 node ec2 dev cluster. I'm trying to setup a snapshot repository for elasticsearch to s3 but I'm getting the following error. AWS command line works for me but setting up through elasticsearch is not. any ideas?

# aws s3 ls s3://d-elasticsearch-snapshots/d-aw2-bdelksa1/
2020-11-25 17:59:58          0 
2020-11-30 17:21:41          5 blah.out

# curl -XPUT "elastic:*****@d-aw2-bdelksa1-1.*****:9200/_snapshot/d-aw2-bdelksa1-repo?pretty" -H 'Content-Type: application/json' -d'
> {
>     "type": "s3",
>     "settings": {
>       "bucket": "d-elasticsearch-snapshots",
>       "client": "default",
>       "base_path": "d-aw2-bdelksa1",
>       "canned_acl": "private",
>       "storage_class": "standard",
>       "role_arn": "arn:aws:iam::695893684697:role/*****-IFX-PowerUser-CrossAccountRole-695893684697",
>       "server_side_encryption": true
>   }
> }'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[d-aw2-bdelksa1-repo] path [d-aw2-bdelksa1] is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[d-aw2-bdelksa1-repo] path [d-aw2-bdelksa1] is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object [d-aw2-bdelksa1/tests-zWEQ0252SU6cPOCWUrdgPw/master.dat] using a single upload",
      "caused_by" : {
        "type" : "amazon_s3_exception",
        "reason" : "amazon_s3_exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 1KFJDR4V1GBN8***; S3 Extended Request ID: QImJhz***/TD9Hm99xh4wVfZYJfABYrxtOaGuolDB6HGwjm7tCBTD7ZQEN0XpUZxFI5ygi*****)"
      }
    }
  },
  "status" : 500
}
2

I've also set my credentials with ...

/usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.access_key
/usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.secret_key
3

any help would be appreciated. i've worked with my aws admin and he is stating that privileges are fine. i'm able to use aws cli to send files there and list but when trying to setup through elastic, getting the Access Denied error.

4

any help?

5

This isn't a valid repository setting so you should remove it. If you want to access S3 using a role, use aws sts assume-role (or equivalent) to obtain temporary credentials and then put those credentials, including the session token, in the keystore instead of your main account credentials.

6

i ended up having a roll tied to my user that has access to the s3 bucket. i also see that whenever you make changes to keystore for s3, you have to run the below to reload them. i ended up deleting the entries i put in and reloading to get it to work.

curl -X POST "localhost:9200/_nodes/reload_secure_settings?pretty"
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.