Elasticsearch | S3 Repository and Snapshot Creation Failure

Elastic Stack Elasticsearch
1

Hi,

I am working on Backup and Restore using s3 on my 4 node cluster.
According to the documentation I first added access_key and secret_key using keystore :
bin/elasticsearch-keystore add s3.client.default.access_key
bin/elasticsearch-keystore add s3.client.default.secret_key

Next, I tried creating repository using Command
PUT http://11.51.1.121:9998/_snapshot/my_s3_repository

{
  "type": "s3",
  "settings": {
    "bucket": "bucket_name"
  }
}

But this very initial stage is failing and an error is throw i.e.,

{
    "error": {
        "root_cause": [
            {
                "type": "amazon_s3_exception",
                "reason": "amazon_s3_exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: AD129AE011109481; S3 Extended Request ID: WOV0JSl3gf+DwUvArwZnEW7D5XvDeRNhjd8TGFJ84JSCTtLfidqDfLv0Uz8s4QiB0hFq0ygxZjc=)"
            }
        ],
        "type": "amazon_s3_exception",
        "reason": "amazon_s3_exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: AD129AE011109481; S3 Extended Request ID: WOV0JSl3gf+DwUvArwZnEW7D5XvDeRNhjd8TGFJ84JSCTtLfidqDfLv0Uz8s4QiB0hFq0ygxZjc=)"
    },
    "status": 500
}

I am using Elasticsearch version 6.4.0
What are the other settings that I am missing, if any?
Or is there any access issues that I am missing here? If so, how can I know what I am missing?

2

Hi,
did you set the grant on your IAM role for your instance?
I often mess this step and get this error. :blush:

3

Thanks gabriel for the response
How do you set the grant on my IAM role for my instances?

4

First result in google:

Hope it can help.

More here:

5

Yes, It sure helps Gabriel!
Could you please tell me what are the ideal Permissions that has to be provided?

6

Here the resource I used to configure mine:

7

Thanks for sharing these links Gabriel
I am not able to download aws-cloud plugin of Elasticsearch.
It says unknown plugin cloud-aws
I am currently using Elasticsearch version 6.4.0
Has these settings differ with my version?

8

The blog post linked above is rather old, and that plugin no longer exists. The reference manual describes how to do this in more recent versions of Elasticsearch.

1 Like
9

Thanks David for your response.

I indeed used the same documentation as I mentioned in my question.
Let me tell you the process I employed.

  1. Installed S3 repository plugin using : sudo bin/elasticsearch-plugin install repository-s3
  2. Added AWS access key settings to the keystore.
    bin/elasticsearch-keystore add s3.client.default.access_key
    bin/elasticsearch-keystore add s3.client.default.secret_key
  3. Created a repository using the following
    PUT _snapshot/my_s3_repository

{
"type": "s3",
"settings": {
"bucket": "my_bucket"
}
}

This is providing me the error,

{
"error": {
"root_cause": [
{
"type": "amazon_s3_exception",
"reason": "amazon_s3_exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: AD129AE011109481; S3 Extended Request ID: WOV0JSl3gf+DwUvArwZnEW7D5XvDeRNhjd8TGFJ84JSCTtLfidqDfLv0Uz8s4QiB0hFq0ygxZjc=)"
}
],
"type": "amazon_s3_exception",
"reason": "amazon_s3_exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: AD129AE011109481; S3 Extended Request ID: WOV0JSl3gf+DwUvArwZnEW7D5XvDeRNhjd8TGFJ84JSCTtLfidqDfLv0Uz8s4QiB0hFq0ygxZjc=)"
},
"status": 500
}

Please tell me what Am I missing here?

10

Is my_bucket the name of your bucket?

11

No.
This is just an example. I have my actual bucket name mentioned under it

12

Have you configured your IAM permissions as per the manual?

13

Thanks David for the response.
My whole problem lied in IAM Permissions
Now the Backup and restore process is working fine.

1 Like
14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.