Create ECS S3 Repository

Elastic Stack Elasticsearch
1

Hi Team

I am using below Code to Register S3 ECS Repository with my ES Cluster.

curl -XPUT "localhost:9200/_snapshot/es_s3_repo?pretty" -H 'Content-Type: application/json' -d'
{
"type": "s3",
"settings": {
"bucket": "es-cmn-bkp-bkt",
"base_path": "snapshots",
"endpoint": "ECS01S3.mywork.com",
"access_key": "mywork-nonprod-s3",
"secret_key": "XXXXXXXXXXXXXXXXXXXXX",
"protocol": "http"
}
}
'

I am facing below ERROR:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "amazon_client_exception",
        "reason" : "amazon_client_exception: Unable to execute HTTP request: es-cmn-bkp-bkt.ECS01S3.mywork.com"
      }
    ],
    "type" : "repository_exception",
    "reason" : "[es_s3_repo] failed to create repository",
    "caused_by" : {
      "type" : "amazon_client_exception",
      "reason" : "amazon_client_exception: Unable to execute HTTP request: es-cmn-bkp-bkt.ECS01S3.mywork.com",
      "caused_by" : {
        "type" : "i_o_exception",
        "reason" : "es-cmn-bkp-bkt.ECS01S3.mywork.com"
      }
    }
  },
  "status" : 500
}

We can see that its using bucket.endpoint to access it "es-cmn-bkp-bkt.ECS01S3.mywork.com"

Can you please help me to get below link , so that I will be able to connect to S3 bucket properly. [ Endpoint/bucket ]

ECS01S3.mywork.com/es-cmn-bkp-bkt

2

It looks like you are using something other than S3 which expects the (now-deprecated) path-style access pattern. If so, you should set path_style_access to true in your client settings.

3

Hi David

Thanks for reply.

Below are the details of my ES Version 5.6.3 and S3 Plugin Details.

{
  "name" : "node-1",
  "cluster_name" : "ES-COMMON",
  "cluster_uuid" : "oatyIujvQUiVTGMTxVvd8A",
  "version" : {
    "number" : "5.6.3",
    "build_hash" : "1a2f265",
    "build_date" : "2017-10-06T20:33:39.012Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}

name   component           version description
node-1 prometheus-exporter 5.6.3.0 Export ElasticSearch metrics to Prometheus
node-2 prometheus-exporter 5.6.3.0 Export ElasticSearch metrics to Prometheus
node-3 prometheus-exporter 5.6.3.0 Export ElasticSearch metrics to Prometheus
node-4 prometheus-exporter 5.6.3.0 Export ElasticSearch metrics to Prometheus
node-5 prometheus-exporter 5.6.3.0 Export ElasticSearch metrics to Prometheus
node-1 repository-s3       5.6.3   The S3 repository plugin adds S3 repositories
node-2 repository-s3       5.6.3   The S3 repository plugin adds S3 repositories
node-3 repository-s3       5.6.3   The S3 repository plugin adds S3 repositories
node-4 repository-s3       5.6.3   The S3 repository plugin adds S3 repositories
node-5 repository-s3       5.6.3   The S3 repository plugin adds S3 repositories

We are using NON-AWS S3 Bucket, i.e ECS S3 Bucket.

As mentioned, I am not able to connect to bucket and register repository.

Hence was thinking and want to try with option "ECS01S3.mywork.com/es-cmn-bkp-bkt" ...

I will execute with your given suggested settings and update in here.

4

Hi David

I suppose this settings is not applicable for ES version 5.6.3.

I am getting below error after setting it in elasticsearch.yml file .

java.lang.IllegalArgumentException: unknown setting [s3.client.default.path_style_access] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

5

Ah, ok, that's quite an old version that's well past the end of its supported life. I suggest upgrading. I don't know off the top of my head whether you can force path-style access in such an old version.

6

Thanks David for your reply.

If we upgrade to ES 7.7 { Latest Version } , will it resolve our error , with our current used Settings in CURL Command ?

Can you please help me in understanding this.

7

Is this behaviour of ES correct ... considering bucketname[.]endpointname ?

8

Hi

I dont know how , but my error type got changed ... Now it say UNKNOWN HOST.

+ curl -XPUT 'localhost:9200/_snapshot/es_s3_repo?pretty' -H 'Content-Type: application/json' '-d
 {
 "type": "s3",
 "settings": {
 "bucket": "es-cmn-bkp-bkt",
 "base_path": "snapshots",
 "endpoint": "ECS01S3.mywork.com",
 "access_key": "mywork-nonprod-s3",
 "secret_key": "XXXXXXXXXXXXXXXXXXXXX",
 "protocol": "http"
 }
 }
 '
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_exception",
        "reason" : "[es_s3_repo] failed to create repository"
      }
    ],
    "type" : "repository_exception",
    "reason" : "[es_s3_repo] failed to create repository",
    "caused_by" : {
      "type" : "amazon_client_exception",
      "reason" : "Unable to execute HTTP request: es-cmn-bkp-bkt.ECS01S3.mywork.com",
      "caused_by" : {
        "type" : "unknown_host_exception",
        "reason" : "es-cmn-bkp-bkt.ECS01S3.mywork.com"
      }
    }
  },
  "status" : 500
}
9

On reflection, a better fix might be to adjust your ECS storage device to support vhost-style access instead of path-style access. I've no idea if this is even possible, you will need to work with their support to figure that out.

An Elasticsearch upgrade should resolve this error, if you set path_style_access to true. Some other adjustments may also be required, check the docs for details. To be clear we do not run any tests against ECS storage devices so there's actually no guarantee it'll work at all. It all depends how faithfully "S3-compatible" your storage device really is.

Yes, that's correct, this is the so-called "vhost-style access" and is how the real S3 works today. This is explained in the link I shared above which also tells you that path-style access is deprecated. This means Amazon may remove support for it from a future SDK version, which would cause your repository to stop working. Unfortunately we don't have a lot of control over when that might happen.

10

Okay. Thanks David for the update and your response.

11

Hi

Anything on this Error Type Error , which could help me ...

12

Hi

curl es-cmn-bkp-bkt.ECS01S3.mywork.com
curl: (6) Could not resolve host: es-cmn-bkp-bkt.ECS01S3.mywork.com; Name or service not known

My Current Java Version :

java -version
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
13

That's not related to elasticsearch I'd say.

es-cmn-bkp-bkt.ecs01s3.mywork.com does not exist in any dns records. You need to fix that I believe or use the IP address instead.

14

Thanks David for your response.

As earlier you have mentioned its using "vhost-style access" and that the method it should be using it.

I am trying to understand what it the reason its not able to resolve as only "ecs01s3.mywork.com" being our endpoint is resolvable from our ES Node successfully. But when bucket name is attached to it for processing by ES , it fails.

Do I need to make this es-cmn-bkp-bkt.ecs01s3.mywork.com entry in /etc/hosts or some settings in elasticsearch.yml file.

15

Because that's how virtual-host-style access works. Please read the link I've shared above (twice). It explains this.

I don't think we can offer much advice on how to configure your storage device to behave like S3 here. A brief google for "ECS S3 virtual-host access" throws up some links that may be helpful. If that doesn't help, I think you will need to ask for support from the supplier of your storage device.

16

Yes , I agreed that you have mentioned about virtual host style access. I have written same in my earlier post.

Thank for all your response and time. It helped to understand the behavior of Path_style_access.

I am closing this thread.

17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.