S3 compatible storage are not able to mounted for snapshot

Elastic Stack Elasticsearch
1

We are trying to mount the s3 storage( S3 compatible DataCore (caringo)) bucket on elastic 9.3.0 using the below command and we are getting "s3_exception: A header you provided implies functionality that is not implemented. (Service: S3, Status Code: 501, Request ID: 286BE1B0D952065D) (SDK Attempt Count: 1)"

PUT \_snapshot/backup-s3-repository?verify=true
{
  "type": "s3",
  "settings": {
    "bucket": "-es-backup",
    "base_path": "elasticsearch-backups",
    "endpoint": "http://swarm.com",
    "protocol": "http",
    "path_style_access": true,
    "access_key": "\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*",
    "secret_key": "\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*",
    "max_retries": 15,
    "disable_chunked_encoding": true,
    "register_operation_disabled": true
  }
}

Also i tried added these parameters in elasticsearch.yml

s3.client.default.path_style_access: true
s3.client.default.read_timeout: 50s

Can someone help whether DataCore (caringo) is supported for S3 backup

2

Hi @Rajesh_Kannan

I suggest you read this in detail

And did you run the following analyzer as suggested?

3

I agree that the analyze repository tool that @stephenb linked would be a good thing to try. It might give you a more useful error message. I suspect that DataCore doesn't support if-none-match headers on a put object request, and that you'll need to set unsafely_incompatible_with_s3_conditional_writes to true for your repository. I've never used DataCore though, so I don't know if that will actually help, or if you'll run into other problems after that.

4

Might be worth noting this option is now marked as deprecated, exactly one minor release after it was Generally Available.

Personally, I'd take very careful note of the wording in the documentation.

(boolean) Elasticsearch uses AWS S3's support for conditional writes to protect against repository corruption. If your repository is based on a storage system which claims to be S3-compatible but does not accept conditional writes, set this setting to true to make Elasticsearch perform unconditional writes, bypassing the repository corruption protection, while you work with your storage supplier to address this incompatibility with AWS S3. Defaults to false.

Are you really working with DataCore on this specific incompatibility with S3 ?

5

I'm thinking this might be a bug in the documentation. The setting is not marked as deprecated in the code. I'm trying to track where this deprecation came from. The name of the setting though definitely implies that this is not our first choice in how we want to interact with a repository.

6

I was wrong. That setting was intentionally deprecated. So be careful using it -- it is subject to be removed in the next major version.