Elasticsearch backup on Object storage S3 Compatibility API

Elastic Stack Elasticsearch
1

Hello everyone,

I have Elasticsearch version 8.12.2 installed on Oracle OCI - OKE , I am trying to configure the snapshot to be uploaded to the Object Storage Amazon S3 Compatibility API (From oracle documentation).

now the access key/secret key configured when i run the API like the following from the Elasticsearch

  curl -X PUT "localhost:9200/_snapshot/bkt-s3-snapshot?pretty" -H 'Content-Type: application/json' -d'
{
"type": "s3",
  "settings": {
    "bucket": "my bucket name",
    "region": "my-oci-region",
    "endpoint": "https://namespace.compat.objectstorage.region.oraclecloud21.com/"
  } 
}
'

I am getting the following error

{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_exception",
        "reason" : "[my bucket name] Could not determine repository generation from root blobs"
      }
    ],
    "type" : "repository_exception",
    "reason" : "[my bucket name] Could not determine repository generation from root blobs",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Exception when listing blobs by prefix [index-]",
      "caused_by" : {
        "type" : "sdk_client_exception",
        "reason" : "sdk_client_exception: The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/"
      }
    }
  },
  "status" : 500
}

I tested the same secret key/access key using python code i wrote and I was able to list and upload the file to the same bucket.

any idea about this ?

2

Any update on this ?

3

Sorry, you asked in the "Search" forum rather than the "Elasticsearch" forum, so your post got fewer eyes on it. I've moved it here.

You want to start with the S3-compatible services section here

The short answer is that many storage systems that claim to be "S3 Compatible" actually mean "You can use the same Rest API as S3 does, and it will mostly look the same ". That's not good enough for Elasticsearch snapshots.

We depend on the exact semantics of S3

  • Which operations are fast and/or cheap to perform
  • Which operations are safe to perform in parallel
  • What guarantees are there about the order in which blobs will be visible to other nodes
  • What each error code means, and how we need to react to it
    etc.

If you have an "S3 compatible" system that doesn't provide those same semantics then it's not sufficiently compatible for Elasticsearch.

I believe other users have had success with Oracle's storage API, but we (Elastic) cannot offer any guarantees about how well it works.

You can try the analysis API which will give a reasonably good indication about whether it works.

4

This indicates that ES is trying to get the access key and secret key from the IMDS, which it will only do if you haven't told it what keys to use explicitly, so it looks like this is not true:

You need to set these keys up.