S3 Snapshot on S3 compatible servers

Elastic Stack Elasticsearch
1

Hi,

I've been trying to setup the snapshot plugin in order to be able to backup my cluster, but I'm running into some issues with setting up the repository in ElasticSearch 7.5.

I'm using OVH OpenStack Swift with S3 compatibility enabled (works perfectly fine for other things like gitlab). I've seen the direct Swift integration for ElasticSearch snapshots but the plugin looks to not be maintained anymore so I would rather go with S3.

So i've setup the access_key and secret_key in the keystore as detailed in the documentation and I'm now trying to setup the snapshot repository but I can't configure it as desired.

PUT /_snapshot/ovh_repository
{
  "type": "s3",
  "settings": {
    "bucket": "elasticsearch-backups",
    "endpoint": "storage.uk.cloud.ovh.net",
    "protocol": "https",
    "path_style_access": true,
    "region": "uk"
  }
}

The issue I'm having is that the region field seems to be ignored when the endpoint is used, therefore even with the configuration above I get the error The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'UK'

As anyone been able to configure the repository in a similar way? Maybe with MinIO or on the IBM cloud?

Thanks

2

Hi @SimonJ

It looks like your problem comes from the fact that we don't support configuring the region in the S3 plugin and you are using an authorisation scheme that uses the region setting.

You could try configuring the region on your server (OpenStack) to us-east-1 to work around this.

That said, we do not officially support OpenStack Swift and Minio may be a better option in terms of maintenance. We are currently running our integration tests for the S3 plugin against Minio so it's pretty much guaranteed to work properly and continue to do so in the future.

3

Hi @Armin_Braun,

Thanks for the quick reply. Unfortunately I can't change the region on the server as it's part of OVH cloud solution.

I had noticed that Swift wasn't officially supported, but having been able to use the S3 compatibility layer previously for other projects I was hoping/expecting this to be a non issue. I could run the snapshot against minio but then I will have to backup the backups so that's not the most practical.

Okay thanks for the reply, I will spend some time to try to find a workaround for this, I will post here with more details if it's useful for anyone else.

Just out of curiosity, how does the s3 plugin deal with MinIO configured with a different region than us-east-1? because I would expect the issue to be pretty much the same as what I'm describing here.

4

@SimonJI think I may have found a workaround for your issue that you can try.

Setting the system property -Daws.region=UK should override the region used by the S3 plugin I think (disclaimer: I did not try this out, but it seems likely to work).

Fair question. So far I have not seen this come up. This may be a result of simply no one having tried using a different region yet. I will investigate this and update here.

2 Likes
5

Thanks for the suggested workaround, it will likely save me quite a bit of time! I will check that later on today and post back the results.

Curious to hear about the MinIO update, I have previously seen people using different MinIO regions to have a one-to-one mapping with the S3 regions they were using.

6

@SimonJ

I was not able to reproduce this issue with Minio. It seems that it always falls back to v2 signing which is not affected by an incorrect region setting a far as I can tell. I looks to me as if the Openstack S3 implementation is forcing v4 auth on the SDK which then breaks due to the region being used for signing.
One thing you should double check is that your S3 credentials are correct. If the SDK fails to authenticate using v2 auth with those it will retry using v4 auth, potentially that's what causing the use of v4.

That said, I opened

to enhance the S3 repository to allow setting the S3 signer region and the signer override and allow supporting a wider range of S3 implementations.

7

@Armin_Braun

Great thanks,

So just a quick update, I've tried to set -Daws.region=UK and that doesn't seem to be changing much, I get the same error (region us-east-1 is wrong) when trying to set the repository.

Thanks for looking at this with MinIO. My credentials are correct, I've been using those credentials for something else and it's been fine, I just double checked with s3cmd and it's also working properly. To add them I used elasticsearch-keystore add s3.client.default.access_key and elasticsearch-keystore add s3.client.default.secret_key as the documentation suggests

However I've noticed that my s3cfg is forcing the signature to use the v2 auth which could be related to the auth version you mentionned:

[default]
host_base = storage.uk.cloud.ovh.net
host_bucket = storage.uk.cloud.ovh.net
bucket_location = uk
use_https = True
signature_v2 = True

access_key = ...
secret_key = ...

enable_multipart = False

From what you're describing it seems that by default v2 auth is used and v4 is used as a fallback; do you know if it's possible to force just v2? Just to try to narrow down the problem.

Thanks very much for the help.

EDIT: Actually I just tried to comment the signature_v2 and it's working properly, so that might not be it

8

@SimonJ

unfortunately I couldn't find any workaround to force v2 auth.

The problem has another step here. V4 would probably work just as well as V2 with your S3 implementation but doesn't because of the broken region setting. So for the S3 CLI client, both auth mechanism work fine because it has the right region set, but ES is broken because it's hard set to us-east-1.

9

Okay I got it working!

The issue purely the region which is good news. What I did was pull the elasticsearch repo, and hardcode the region in the EndpointConfiguration, rebuild the plugin and installed it in a custom Docker image

The "patch"

diff --git a/plugins/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3Service.java b/plugins/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3Service.java
index 3ba8145f6a4..d502f4a2d77 100644
--- a/plugins/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3Service.java
+++ b/plugins/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3Service.java
@@ -151,7 +152,7 @@ class S3Service implements Closeable {
         //
         // We do this because directly constructing the client is deprecated (was already deprecated in 1.1.223 too)
         // so this change removes that usage of a deprecated API.
-        builder.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(endpoint, null));
+        builder.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(endpoint, "UK"));
         if (clientSettings.pathStyleAccess) {
             builder.enablePathStyleAccess();
         }

And the Dockerfile

FROM docker.elastic.co/elasticsearch/elasticsearch:7.5.2

COPY repository-s3-7.5.2-SNAPSHOT.zip /tmp
RUN bin/elasticsearch-plugin install --batch file:///tmp/repository-s3-7.5.2-SNAPSHOT.zip

Now creating the repository works fine

PUT /_snapshot/ovh_repository
{
  "type": "s3",
  "settings": {
    "bucket": "elasticsearch-backups",
    "endpoint": "storage.uk.cloud.ovh.net",
    "protocol": "https",
    "path_style_access": true,
    "disable_chunk_encoding": true
  }
}

{"acknowledged":true}
1 Like
10

Thanks for confirming the fix @SimonJ , good to know that the ability to specify the region will suffice to fix this issue :slight_smile:

11

@SimonJ

FYI I just merged

which will allow setting region and signing algorithm overrides starting in 7.7.

12

Thanks very much Armin, looking forward to use that instead of my current hacky workaround!

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.