S3 repository plugin: Is it possible to restore a snapshot from a non-read-only repository using a read-only access_key?

Elastic Stack Elasticsearch
1

Hello all,

I am trying to restore a snapshot from an s3 bucket created with Elasticsearch 2.2.1, using Elasticsearch 5.0.2. The bucket is not read-only, but the access_key for AWS only has read-only access.

When I try to restore a snapshot using:

POST http://localhost:9200/_snapshot/repo_name/snapshot_name/_restore
{
  "index_settings": {
    "index.number_of_replicas": 0
  }
}

It produces the following error:

{
  "error": {
    "root_cause": [
      {
        "type": "repository_exception",
        "reason": "[repo_name] failed to update the repository index blob with indices data on startup"
      }
    ],
    "type": "repository_exception",
    "reason": "[repo_name] failed to update the repository index blob with indices data on startup",
    "caused_by": {
      "type": "i_o_exception",
      "reason": "Unable to upload **** (anonymized this),
      "caused_by": {
        "type": "amazon_s3_exception",
        "reason": "Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 1AE8CE8E63CB4BB8)"
      }
    }
  },
  "status": 500
}

If I search for that particular error message in the source code, it seems like this exception is thrown when the repository is not read-only and it tries to update some information in that repository, which of course is not allowed if you have an AWS access_key with read-only access.

So my question is: Is there a way to restore that snapshot using a read-only AWS access_key? Or do I have to have an AWS access_key with write access?

2

Try to create the repo with readonly instead of read_only. There is a bug in the docs.

3

The repo is not read-only, and is supposed to be that. It is also hosted by another company, so I cannot change it even if I wanted to.

My question still remains: can you do a read-only snapshot restore on a non-read-only repo?

4

How do you create the repository in your elasticsearch cluster?

5

I create the repo in my cluster like this (using fake names in this example):

PUT http://localhost:9200/_snapshot/repo_name?verify=false
{
  "type": "s3",
  "settings": {
    "bucket": "bucket_name",
    "access_key": "*******",
    "secret_key": "*******",
    "base_path": "some/path",
    "compress": "true",
  }
}

I am aware that I use verify=false. I need to, because verification will fail for the same reason, which is that my AWS access_key is read-only. In version 2.X however, the snapshot restore would still work. In version 5.0.2, I can't get it to work, that's why I am asking if there is something I am missing, or if I am running into a (new or old) limitation of the snapshot restore.

6

Can you try:

PUT http://localhost:9200/_snapshot/repo_name?verify=false
{
  "type": "s3",
  "settings": {
    "bucket": "bucket_name",
    "access_key": "*******",
    "secret_key": "*******",
    "base_path": "some/path",
    "compress": "true",
    "readonly": true
  }
}
7

I was just doing that, out of curiosity, and that is indeed what I was looking for. I just was misunderstanding how a repository works in elasticsearch. I thought it was just a pointer to the bucket in S3 (which is hosted by the other company), but I am actually creating a bit more than that in my own cluster, or so it seems.

Thanks David, for your help!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.