Repository-s3 plugin does not support S3-compatible services as documentation says

Hi is there anyone who achieve configuring repository-s3 plugin to work with S3-compatible service like CEPH. I completely follow the documentation (and many forums) and try to configure it but every time I failed. I did following things:

I installed repository-s3 plugin:

bin/elasticsearch-plugin install --batch repository-s3

I set appropriate parameters in elasticsearch.yml config file (on all of my tree nodes) and restart each node:

s3:
  client:
    default:
      region: "default"
      max_retries: 3
      protocol: "https"
      endpoint: "<my_ceph_endpoint>"
      signer_override: "S3SignerType"
      read_timeout: "180s"

I add secret key and access key to keystore (on all of my tree nodes) as documentation says:

echo <access_key> | bin/elasticsearch-keystore add  --stdin s3.client.default.access_key
echo <secret_key> | bin/elasticsearch-keystore add  --stdin s3.client.default.secret_key

I reload secure settings (on all of my tree nodes):

curl -u <user>:<password> -X POST "https://<elasticsearch_url>:9200/_nodes/reload_secure_settings"

Then I try to create snapshot repository (bucket exists and was created with usage of the same access and secret keys):

curl -u <user>:<password> -X PUT https://<elasticsearch_url>:9200/_snapshot/test1 -H 'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
    "client": "default",
    "bucket": "<bucket_name>"
  }
}'

No matter what I do I always get response like this:

{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[test1] path  is not accessible on master node"}],"type":"repository_verification_exception","reason":"[test1] path  is not accessible on master node","caused_by":{"type":"i_o_exception","reason":"Unable to upload object [tests-Ds-i_M2TRf2fl5D1W4ovMg/master.dat] using a single upload","caused_by":{"type":"sdk_client_exception","reason":"The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/"}}},"status":500}

It seems that it always asks this endpoint http://169.254.169.254/latest/meta-data/iam/security-credentials/ but my instances are not AWS (they are on OpenStack).

Can anyone explain me what happens here? Is there anything that I missed? Or it is simply impossible with current repository-s3 plugin and documentation is faulty?

I tried it on Elasticsearch 7.7.0 and on 7.10.0 versions

It's definitely possible, but if your cluster is accessing http://169.254.169.254/latest/meta-data/iam/security-credentials/ then you have not configured the credentials in the keystore correctly on all nodes.

Thank you very much that tip was helpful, my script which saves that keys in keystore was somewhere faulty

1 Like