Elasticsearch creating snapshots to gce with the ability to put it into a folder

Elastic Stack Elasticsearch
#1

Hello,

I am using elasticsearch v5 and I am trying to create snapshots to the gce storage. I have a bucket created for all kind of backups that is why I would like to puthe elasticsearch backups into their own folder.

If I use the following command everything is OK but the indexes are placed into the root of the bucket

curl -XPUT http://localhost:19270/_snapshot/test?pretty -H 'Content-Type: application/json' -d'
{
  "type": "gcs",
  "settings": {
    "bucket": "gce_snapshots",
    "service_account": "service_account.json"
  }
}
'

I have tried precreating the folders on the gce bucket and use it like this:

curl -XPUT http://localhost:19270/_snapshot/test?pretty -H 'Content-Type: application/json' -d'
{
  "type": "gcs",
  "settings": {
    "bucket": "gce_snapshot/test",
    "service_account": "service_account.json"
  }
}
'

but I get the following error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "blob_store_exception",
        "reason" : "Unable to check if bucket [gce_snapshot/test] exists"
      }
    ],
    "type" : "repository_exception",
    "reason" : "[test] failed to create repository",
    "caused_by" : {
      "type" : "blob_store_exception",
      "reason" : "Unable to check if bucket [gce_snapshot/test] exists",
      "caused_by" : {
        "type" : "google_json_response_exception",
        "reason" : "400 Bad Request\n{\n  \"code\" : 400,\n  \"errors\" : [ {\n    \"domain\" : \"global\",\n    \"message\" : \"Invalid bucket name: 'gce_snapshot/test'\",\n    \"reason\" : \"invalid\"\n  } ],\n  \"message\" : \"Invalid bucket name: 'snapshots.wpcloud.io/test'\"\n}"
      }
    }
  },
  "status" : 500
}

Any idea how to achieve this?

#2

Did you install the ES GCE plugin?

https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-gcs.html

#3

Of course I did, only the information the url gives does not tell how to take a snapshot into a folder on the bucket which I need.

#4

Didn't mean to ask the obvious , but my job habits tend to creep up on me. Does the snapshot work if you didn't specify the "test" folder?

Nevermind, I see that you said it did work.

#5

As I previously said, the snapshot works if the folder is not specified but I need to do that cause we are taking snapshots on different elasticsearch clusters for different customers. Therefore each needs to have their own place without the interference of other customers.

#6

From the code it looks like it will check for privileges to create the folder. If you do have that setup I would restarting ES then issue the snapshot command again.

  104     boolean doesBucketExist(String bucketName) {
  105         try {
  106             return doPrivileged(() -> {
  107                 try {
  108                     Bucket bucket = client.buckets().get(bucketName).execute();
  109                     if (bucket != null) {
  110                         return Strings.hasText(bucket.getId());
  111                     }
  112                 } catch (GoogleJsonResponseException e) {
  113                     GoogleJsonError error = e.getDetails();
  114                     if ((e.getStatusCode() == HTTP_NOT_FOUND) || ((error != null) && (error.getCode() == HTTP_NOT_FOUND))) {
  115                         return false;
  116                     }
  117                     throw e;
  118                 }
  119                 return false;
  120             });
  121         } catch (IOException e) {
  122             throw new BlobStoreException("Unable to check if bucket [" + bucketName + "] exists", e);
  123         }
  124     }
#7

What kind of privileges do I need for that? I have the service user with write access to the bucket. I didn't see any other privileges possible but read or write. Also how to mention that I want to put the snapshots into xxx folder?

#8

I'd do:

curl -XPUT http://localhost:19270/_snapshot/test?pretty -H 'Content-Type: application/json' -d'
{
  "type": "gcs",
  "settings": {
    "bucket": "gce_snapshot",
    "service_account": "service_account.json",
    "base_path": "test"
  }
}
'
#9

This is unrelated. doPriviliged() has been introduced recently for other reasons.

#10

Perfect. As always you get me to the right direction. Thank you very much

#11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.