Repository-s3 plugin does not support S3-compatible services as documentation says

Elastic Stack Elasticsearch
1

Hi is there anyone who achieve configuring repository-s3 plugin to work with S3-compatible service like CEPH. I completely follow the documentation (and many forums) and try to configure it but every time I failed. I did following things:

I installed repository-s3 plugin:

bin/elasticsearch-plugin install --batch repository-s3

I set appropriate parameters in elasticsearch.yml config file (on all of my tree nodes) and restart each node:

s3:
  client:
    default:
      region: "default"
      max_retries: 3
      protocol: "https"
      endpoint: "<my_ceph_endpoint>"
      signer_override: "S3SignerType"
      read_timeout: "180s"

I add secret key and access key to keystore (on all of my tree nodes) as documentation says:

echo <access_key> | bin/elasticsearch-keystore add  --stdin s3.client.default.access_key
echo <secret_key> | bin/elasticsearch-keystore add  --stdin s3.client.default.secret_key

I reload secure settings (on all of my tree nodes):

curl -u <user>:<password> -X POST "https://<elasticsearch_url>:9200/_nodes/reload_secure_settings"

Then I try to create snapshot repository (bucket exists and was created with usage of the same access and secret keys):

curl -u <user>:<password> -X PUT https://<elasticsearch_url>:9200/_snapshot/test1 -H 'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
    "client": "default",
    "bucket": "<bucket_name>"
  }
}'

No matter what I do I always get response like this:

{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[test1] path  is not accessible on master node"}],"type":"repository_verification_exception","reason":"[test1] path  is not accessible on master node","caused_by":{"type":"i_o_exception","reason":"Unable to upload object [tests-Ds-i_M2TRf2fl5D1W4ovMg/master.dat] using a single upload","caused_by":{"type":"sdk_client_exception","reason":"The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/"}}},"status":500}

It seems that it always asks this endpoint http://169.254.169.254/latest/meta-data/iam/security-credentials/ but my instances are not AWS (they are on OpenStack).

Can anyone explain me what happens here? Is there anything that I missed? Or it is simply impossible with current repository-s3 plugin and documentation is faulty?

I tried it on Elasticsearch 7.7.0 and on 7.10.0 versions

2

It's definitely possible, but if your cluster is accessing http://169.254.169.254/latest/meta-data/iam/security-credentials/ then you have not configured the credentials in the keystore correctly on all nodes.

1 Like
3

Thank you very much that tip was helpful, my script which saves that keys in keystore was somewhere faulty

2 Likes
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.