ECE: Add S3 Repository Using Instance Roles

Elastic Orchestration Elastic Cloud Enterprise (ECE)
1

Using a single-host ECE 3.5 environment, we're attempting to connect a deployment to an S3 bucket. The AWS policy is configured to allow access to the S3 bucket from the instance hosting ECE using an instance role.

We've tested connectivity/access from the ECE host to the S3 bucket using the aws s3 ls command.

The ECE Admin UI (Platform -> Repositories -> Add Repository) requires an Access Key and a Secret Key, but if you use the API, you can get ECE to accept the settings for a repository without those fields (this was suggested by Elastic support).

... and we added s3.client.default.endpoint as a setting to the elasticsearch settings for the deployment.

However, when we go to the deployment and navigate Stack Management -> Snapshot and Restore -> Repositories and then click on found-snapshots and click the "Verify Repository", we get this message:

{
  "name": "ResponseError",
  "message": "repository_verification_exception\n\tCaused by:\n\t\ti_o_exception: Unable to upload object [snapshots/8a31846b<redacted>d254a/tests-9AqsGB<redacted>94A/master.dat] using a single upload\n\tRoot causes:\n\t\trepository_verification_exception: [found-snapshots] path [snapshots/8a31846b<redacted>254a] is not accessible on master node"
}

A test was also conducted without the s3.client.default.endpoint setting in the elasticsearch config and the results were the same.

Has anyone been able to get an S3 repository to work in an ECE-managed environment? I'd be content with a declarative statement in the vendor documentation saying that using instance roles to access S3 buckets is not supported with ECE, but we've been unable to get concurrence on that either. Any ideas?

2

Hello "billfischermnit"

We use on-prem ECE and used these settings since version 3.4.1

Never tried S3 with anonymous access, but got it working for both regular snapshots aka "found-snapshots", and for "frozen" Searchable snapshot, with the use of the access_key and secret_key.

The "no_verify_ssl" is needed if your plattform/deployments do not trust the s3.

Plattformwize: ("found-snapshot")
plattform/repository/advanced

Add S3 "client_settings" reository for found-snapshots

{
  "type": "s3",
  "settings": {
    "bucket": "name_of_the_bucket_in_S3",
    "endpoint": "fqdn.or.ip.of.server:port",
    "no_verify_ssl": "true",
    "access_key": "This_is_key_for_the_s3_bujcket",
    "secret_key": "SuperSecureKeyWithLotsOfCharacters@Numbers",
    "path_style_access": "true",
    "protocol": "https",
    "region": "My country code"
  }
}

Name of client has nothing to do with s3 bucket, so this can be whatever

Deploymentwize:
API:

PUT /_snapshot/name_of_the_repository
{
"type": "s3",
"settings": {
"bucket" : "name_of_the_bucket",
"client" : "Name_of_the_client", (same as above)
"no_verify_ssl": "true",
"path_style_access" : "true"
}
}

Elasticsearch user settings and extensions

s3.client.Name_of_the_client.endpoint: "fqdn.or.ip.of.server:port"

elasticsearch keystore in security

Add 2 keys

s3.client.Name_of_the_client.access_key
s3.client.Name_of_the_client.secret_key

Hope this helps