Oracle cloud S3 repository - No Snapshots after restart

Elastic Stack Elasticsearch
1

Hi there,

I have a problem with the snapshot API when it comes to use it with a S3 repository on oracle cloud.

My elasticsearch 7.5.2 instance is running inside a docker container.

GET /_snapshot/s3_repository/

returns my repository settings

{
  "s3_repository" : {
    "type" : "s3",
    "settings" : {
      "bucket" : "xxxxxxxxxx",
      "base_path" : "xxxxxxxxxxx",
      "endpoint" : "xxxxxxxxxx.compat.objectstorage.eu-frankfurt-1.oraclecloud.com",
      "protocol" : "https",
      "region" : "eu-frankfurt-1"
    }
  }
}

After that I start the backup via

PUT /_snapshot/s3_repository/snapshot-test
{
  "indices": "test"
}

Now I can see my saved snapshots

GET /_snapshot/s3_repository/_all

which returns

{
  "snapshots" : [
    {
      "snapshot" : "snapshot-test",
      "uuid" : "dKowu34RSDmSMoPRJPgH7g",
      "version_id" : 7050299,
      "version" : "7.5.2",
      "indices" : [
        "test"
      ],
      "include_global_state" : true,
      "state" : "SUCCESS",
      "start_time" : "2020-02-07T14:35:44.556Z",
      "start_time_in_millis" : 1581086144556,
      "end_time" : "2020-02-07T14:35:45.156Z",
      "end_time_in_millis" : 1581086145156,
      "duration_in_millis" : 600,
      "failures" : [ ],
      "shards" : {
        "total" : 1,
        "failed" : 0,
        "successful" : 1
      }
    }
  ]
}

The next step shows the problem. After restarting the elasticsearch docker container I still see my repo but the saved snapshots list is empty

GET /_snapshot/s3_repository/_all

{
  "snapshots" : []
}

What's the problem here? Did I miss something?

2

Hi @Alrik_Geselle,

what's the physical contents of the path of your repository in the bucket? Is there any snap- or meta- prefix blobs in there?

3

Hi @Armin_Braun,

yes there meta and snapshot files. But there is no subfolder named "s3_repository". All files are directly in the storage path. I think this will be problematic when I want to use my oracle S3 storage for a second repository.

4

If you want your data to go to the subfolder "s3_repository" you'll have to set

  "base_path" : "s3_repository",

in your repository setting when you create the repo. If you don't set anything for the base path, then the root of the bucket will be used. Could it be that you used a tricky base path like simply / or so that the Oracle S3 implementation may not properly support? (that's my best guess so far).
Ideally, I'd try just using "s3_repository" as the base path without any slashes or so and see if that solves things to start with and then try and narrow down the issue if that helped.

5

Thank you for your answer.

As you can see, I already set a base path to my repo settings but still the root will be used.
I tested it again with an Amazon S3 repo and it works. So I think it must be an Oracle S3 problem. Is there anything you guys can do to locate/fix this problem?

6

This is really strange. The way I've seen this happen so far with non-AWS S3 is when using a root path that contained / or some other special characters that led to the URL not being properly decoded.

Is there anything you guys can do to locate/fix this problem?

Not really I'm afraid. This seems to be something to contact Oracle cloud support about to me. There simply seems to be some non-S3-compliant behaviour here on their end that is breaking the S3 SDK. That's not something we can work around I'm afraid.

7

Ok thank you very much. We will contact our Oracle Cloud service partner.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.