Snapshots are failing in ECE minio

[logger (Malte Leek)]

Hi there,

I have integrated a repository with minio. After I recognized that elasticsearch 6.x was not able to resolve hostnames I changed to IP. Elasticsearch 7.x worked fine for a while.

I use version 2.4.1 and updated yesterday from 2.3.1. But the error has already been in 2.3.1.
Now one deployment has some error by creating a snapshot. But it has 100 working snapshots already:

Unknown s3 client name [cloud-REPONAME]. Existing client configs: Default

In the advanced settings I see:

  "snapshot": {
    "enabled": true,
    "repository": {
      "config": {
        "repository_id": "REPONAME",
        "snapshot_config_type": "reference"
      }
    },
    "suspended": {}
  }

Why is in the Error message a "cloud-" prefix?

[Alex_Piggott]

From 7.x/2.4 (not 100% what the version boundary is), the snapshot config code is distributed between 2 places - the _snapshot endpoint, and the secure settings

It looks from the error like the _snapshot endpoint includes the auto-generated client field (you can confirm by doing GET _snapshot/found-snapshots), but the cluster does not include the secure settings (eg you won't find the S3 settings in the secure settings in the advanced cluster data lower down)

Can you post (sanitized versions of) those two JSON object?

Can you try applying a no-op plan to the cluster?

[logger (Malte Leek)]

{
  "found-snapshots": {
    "type": "s3",
    "settings": {
      "bucket": "ece-es7",
      "client": "cloud-REPONAME",
      "base_path": "snapshots/c371465ec830483d8a640e8cb49e3b",
      "endpoint": "https://IPMINIO:9000",
      "protocol": "https"
    }
  }
}

This is the only Information in advanced I found:

  "snapshot": {
    "enabled": true,
    "repository": {
      "config": {
        "repository_id": "REPONAME",
        "snapshot_config_type": "reference"
      }
    },
    "suspended": {}
  },

[logger (Malte Leek)]

I do not find the secure settings. This could be the error.

How do I get the settings set on default?

[dimatkach (Dima)]

Could you show the repo definition from Platform/Repositories (switch "Repository Type" to Advanced to get to the actual json). Remove the values for the secrets, I just want to see that the keys are there, and look right.

[logger (Malte Leek)]

I can do this when I am back in office on Tuesday.

But it is the same from the docs:
https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configuring-minio.html

 {
     "type": "s3",
      "settings": {
         "bucket": "ece-backup",
         "access_key": "<your Minio AccessKey>",
         "secret_key": "<your Minio SecretKey>",
         "endpoint": "https://IP>:9000",
         "protocol": "https"
      }
  }

thanks it advance.

[logger (Malte Leek)]

Hi there,

as I have already written. The configuration is:

{
  "settings": {
    "access_key": "ACCESS_KEY",
    "bucket": "ece-backup",
    "endpoint": "https://IP:9000",
    "protocol": "https",
    "secret_key": "SECRETKEY"
  },
  "type": "s3"
}

[dimatkach (Dima)]

Hi,

Could you just clarify ... in your last post the name was "bucket": "ece-backup" ,
but you had posted _snapshot/found-snapshots is returning "bucket": "ece-es7" ..
are there multiple repos?

[logger (Malte Leek)]

Hi,

It is "bucket": "ece-es7". I just changed it to the same like the example.

We have multiple repos. One for Elasticsearch 6.x and one for Elasticsearch 7.x.

[dimatkach (Dima)]

So, the way this is supposed to work is that when ES container is allocated, we take the secrets from your shared repo definition and add them to the Elasticsearch keystore on each node. Looks like this did not happen in your case for some reason.

One more thing to try is to go to that repo definition under Platform/Repositories, switch to advanced, and (after making sure the settings are correct), Save it. That should cause the credentials to be written to keystore again in case it got out of sync somehow.

After saving, wait a few minutes to give it time to sync to all nodes, then from the API console do

PUT _snapshot/found_snapshots
{
    "type": "s3",
    "settings": {
      "bucket": "ece-es7",
      "client": "cloud-REPONAME",
      "base_path": "snapshots/c371465ec830483d8a640e8cb49e3b",
      "endpoint": "https://IPMINIO:9000",
      "protocol": "https"
    }
 }

(replacing "REPONAME" and "IPMINIO" with the real things of course).
If that works, it should be all fixed.

If it fails though, it seems that we would have to take a closer look at your configuration.
In that case, would it be possible for you to open a support ticket to make it easier to exchange information? We would then circle back and post the result here once we understand what is going on?

I hope this helps!