Getting error while configuring mcqueen as a S3 alternative for storing snapshots for elasticsearch

Elastic Stack Elasticsearch
1

I'm trying to configure mcqueen for storing elasticsearch snapshots.
I have created a bucket ./s3curl.pl --id=test --createBucket -- http://store-test.blobstore.xxx.com/snapshots

But when I'm trying to create a repository, it's failing with below error-

curl -XPUT 'localhost:9200/_snapshot/s3' -H "Content-Type: application/json" -d ' { "type": "s3", "settings": { "bucket": "snapshots" } }'

{"error":{"root_cause":[{"type":"sdk_client_exception","reason":"sdk_client_exception: Unable to execute HTTP request: snapshots.store-test.blobstore.xxx.com"}],"type":"repository_exception","reason":"[s3] failed to create repository","caused_by":{"type":"sdk_client_exception","reason":"sdk_client_exception: Unable to execute HTTP request: snapshots.store-test.blobstore.xxx.com","caused_by":{"type":"i_o_exception","reason":"snapshots.store-test.blobstore.xxx.com"}}},"status":500}

If you see in error, the url it is trying to access is "snapshots.store-test.blobstore.xxx.com" but instead it should access at "store-test.blobstore.xxx.com/snapshots"

My configuration for elasticsearch.yml on all nodes is-

s3.client.default.endpoint: store-test.blobstore.xxx.com
s3.client.default.protocol: http

and have stored access_key and secret_key in keystore of all nodes. Also restarted Elasticsearch after storing keystore

2

Hi Team,

Can I get any response on this ? I know I have to be patient but it's been 7 days since I put up the issue. I tried few things but still stuck at this error. Any help will be appreciated.
Thanks

3

Is snapshots.store-test.blobstore.xxx.com a S3 repository or something compatible?

Your best chance to find the root cause of it is I think to debug. Change the log level to DEBUG for org.elasticsearch.repositories and com.amazonaws.

5

As we don't support it, your best chance is to change the log level and see what is happening.

Which elasticsearch version are you using?

6

ES - 6.3.2

7

Worth updating to 6.8.0. I think that some improvements have been done in the meantime so there might be a chance that it fixes your problem.
But again, logs would help.

8

Hi,

I dont know mcqueen but I know S3.
The way you have your mcqueen setup right now makes it NOT compatible with S3.

S3 supports BOTH virtual host style bucket access AND path style bucket access. Your S3 compatible service ONLY supports path style bucket access because the error you have is because the plugin tries to access your bucket with the virtual host style and it doesn't work... you're not setup for it.

Because this plugin uses the AWS SDK and the S3 client from it, it really assumes that it is talking to S3 and you really have to be 100% compatible with S3.

As far as I understand it, the plugin s3-repository from Elastic doesn't expose the SDK setting for "addressing_style" which can be used to force the s3 client to use only path style bucket access.
So I don't think you can fix this without Elastic improving the plugin so that it can be configured to force usage of path style.

In the absence of this, my best guest, you'll have to try it, is to use a bucket name that is NOT DNS compliant, which should trigger the SDK to use path style bucket access because it can't use virtual host style anymore.

Make a bucket with an underscore in the name: my_bucket
Test to make sure it does still work with your mcqueen. By accessing it manually or with some other tool you use normally.
Then config ES to use that bucket instead of the one called "snapshots" which does not work for you.

If it doesn't work post the same information you posted initially here about your issue.

Also maybe check if you can make your mcqueen setup REALLY compatible with S3... which means it needs to support BOTH virtual host style access for buckets AND path style access. That would also fix your issue.

You can also open a feature request on github, ref this post, explain your problem and ask if s3-repository plugin could expose the S3 client setting to change the "addressing_style". If it was exposed that should also fix your issue.

Last but not least I'll add that path style access to buckets in S3 is slowly dying and going away. S3 stopped accepting the creation of new buckets with DNS non compliant names, which means virtual host style access always works for those buckets. And they also plan to completely stop supporting path style access.
Which means that you NEED to make mcqueen support this before long, or it will be left out of S3 compatibility completely.

Let us know if you try with a new bucket with an underscore as a workaround.

9

It used to be the case in the past. AFAIK it will come back in the future. See

1 Like
10

I should have checked github correctly instead of just the current master branch for the plugin code :slight_smile:

Thanks @dadoonet with 2 brains we're killing it! :exploding_head:

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.