Elasticsearch creating snapshots to gce with the ability to put it into a folder

Elastic Stack Elasticsearch
1

Hello,

I am using elasticsearch v5 and I am trying to create snapshots to the gce storage. I have a bucket created for all kind of backups that is why I would like to puthe elasticsearch backups into their own folder.

If I use the following command everything is OK but the indexes are placed into the root of the bucket

curl -XPUT http://localhost:19270/_snapshot/test?pretty -H 'Content-Type: application/json' -d'
{
  "type": "gcs",
  "settings": {
    "bucket": "gce_snapshots",
    "service_account": "service_account.json"
  }
}
'

I have tried precreating the folders on the gce bucket and use it like this:

curl -XPUT http://localhost:19270/_snapshot/test?pretty -H 'Content-Type: application/json' -d'
{
  "type": "gcs",
  "settings": {
    "bucket": "gce_snapshot/test",
    "service_account": "service_account.json"
  }
}
'

but I get the following error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "blob_store_exception",
        "reason" : "Unable to check if bucket [gce_snapshot/test] exists"
      }
    ],
    "type" : "repository_exception",
    "reason" : "[test] failed to create repository",
    "caused_by" : {
      "type" : "blob_store_exception",
      "reason" : "Unable to check if bucket [gce_snapshot/test] exists",
      "caused_by" : {
        "type" : "google_json_response_exception",
        "reason" : "400 Bad Request\n{\n  \"code\" : 400,\n  \"errors\" : [ {\n    \"domain\" : \"global\",\n    \"message\" : \"Invalid bucket name: 'gce_snapshot/test'\",\n    \"reason\" : \"invalid\"\n  } ],\n  \"message\" : \"Invalid bucket name: 'snapshots.wpcloud.io/test'\"\n}"
      }
    }
  },
  "status" : 500
}

Any idea how to achieve this?

2

Did you install the ES GCE plugin?

https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-gcs.html

3

Of course I did, only the information the url gives does not tell how to take a snapshot into a folder on the bucket which I need.

4

Didn't mean to ask the obvious , but my job habits tend to creep up on me. Does the snapshot work if you didn't specify the "test" folder?

Nevermind, I see that you said it did work.

5

As I previously said, the snapshot works if the folder is not specified but I need to do that cause we are taking snapshots on different elasticsearch clusters for different customers. Therefore each needs to have their own place without the interference of other customers.

6

From the code it looks like it will check for privileges to create the folder. If you do have that setup I would restarting ES then issue the snapshot command again.

  104     boolean doesBucketExist(String bucketName) {
  105         try {
  106             return doPrivileged(() -> {
  107                 try {
  108                     Bucket bucket = client.buckets().get(bucketName).execute();
  109                     if (bucket != null) {
  110                         return Strings.hasText(bucket.getId());
  111                     }
  112                 } catch (GoogleJsonResponseException e) {
  113                     GoogleJsonError error = e.getDetails();
  114                     if ((e.getStatusCode() == HTTP_NOT_FOUND) || ((error != null) && (error.getCode() == HTTP_NOT_FOUND))) {
  115                         return false;
  116                     }
  117                     throw e;
  118                 }
  119                 return false;
  120             });
  121         } catch (IOException e) {
  122             throw new BlobStoreException("Unable to check if bucket [" + bucketName + "] exists", e);
  123         }
  124     }
7

What kind of privileges do I need for that? I have the service user with write access to the bucket. I didn't see any other privileges possible but read or write. Also how to mention that I want to put the snapshots into xxx folder?

8

I'd do:

curl -XPUT http://localhost:19270/_snapshot/test?pretty -H 'Content-Type: application/json' -d'
{
  "type": "gcs",
  "settings": {
    "bucket": "gce_snapshot",
    "service_account": "service_account.json",
    "base_path": "test"
  }
}
'
9

This is unrelated. doPriviliged() has been introduced recently for other reasons.

10

Perfect. As always you get me to the right direction. Thank you very much

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.