AWS S3 repository for snapshot/restore in elasticsearch

Elastic Stack Elasticsearch
1

I want to use AWS S3 bucket for Elasticsearch snapshot/restore of indices.

I have read the official doc but I am unable to understand what all properties will be needed in my elasticsearch.yml file to connect to my S3 bucket.

We have a ecosystem where I can order S3 bucket and have access & secret key.

image
image1336×580 64.5 KB

This the error I am getting:

{
  "name": "ResponseError",
  "meta": {
    "body": {
      "error": {
        "root_cause": [
          {
            "type": "repository_verification_exception",
            "reason": "[IST_ELASTIC_BKP] path  is not accessible on master node"
          }
        ],
        "type": "repository_verification_exception",
        "reason": "[IST_ELASTIC_BKP] path  is not accessible on master node",
        "caused_by": {
          "type": "i_o_exception",
          "reason": "Unable to upload object [tests-KWtjRD-oS-qOGKiigGKqVg/master.dat] using a single upload",
          "caused_by": {
            "type": "sdk_client_exception",
            "reason": "Failed to connect to service endpoint: ",
            "caused_by": {
              "type": "socket_timeout_exception",
              "reason": "Connect timed out"
            }
          }
        }
      },
      "status": 500
    },
    "statusCode": 500,
    "headers": {
      "x-opaque-id": "d9e795d3-6d23-4795-8e87-ab14810be355;kibana:application:management:",
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json;charset=utf-8",
      "content-length": "571"
    },
    "meta": {
      "context": null,
      "request": {
        "params": {
          "method": "POST",
          "path": "/_snapshot/IST_ELASTIC_BKP/_verify",
          "querystring": "",
          "headers": {
            "user-agent": "Kibana/8.6.2",
            "x-elastic-product-origin": "kibana",
            "authorization": "Basic ZWxhc3RpYzpGWXFVVDIySkJEaTlIQ3pZUkEqNw==",
            "x-opaque-id": "d9e795d3-6d23-4795-8e87-ab14810be355;kibana:application:management:",
            "x-elastic-client-meta": "es=8.4.0p,js=16.18.1,t=8.2.0,hc=16.18.1",
            "accept": "application/vnd.elasticsearch+json; compatible-with=8,text/plain"
          }
        },
        "options": {
          "opaqueId": "d9e795d3-6d23-4795-8e87-ab14810be355;kibana:application:management:",
          "headers": {
            "x-elastic-product-origin": "kibana",
            "user-agent": "Kibana/8.6.2",
            "authorization": "Basic ZWxhc3RpYzpGWXFVVDIySkJEaTlIQ3pZUkEqNw==",
            "x-opaque-id": "d9e795d3-6d23-4795-8e87-ab14810be355",
            "x-elastic-client-meta": "es=8.4.0p,js=16.18.1,t=8.2.0,hc=16.18.1"
          }
        },
        "id": 1
      },
      "name": "elasticsearch-js",
      "connection": {
        "url": "https://10.107.94.218:9090/",
        "id": "https://10.107.94.218:9090/",
        "headers": {},
        "status": "alive"
      },
      "attempts": 0,
      "aborted": false
    },
    "warnings": null
  }
}
2

If you are running on premise, you need to run that on every node:

bin/elasticsearch-keystore add s3.client.default.access_key
bin/elasticsearch-keystore add s3.client.default.secret_key

My 2 cents

3

I have added these but received following error:

{
  "name": "ResponseError",
  "meta": {
    "body": {
      "error": {
        "root_cause": [
          {
            "type": "repository_verification_exception",
            "reason": "[IST_ELASTIC_BKP] path  is not accessible on master node"
          }
        ],
        "type": "repository_verification_exception",
        "reason": "[IST_ELASTIC_BKP] path  is not accessible on master node",
        "caused_by": {
          "type": "i_o_exception",
          "reason": "Unable to upload object [tests-6yMMVyGRToqFrs6AiobhTQ/master.dat] using a single upload",
          "caused_by": {
            "type": "sdk_client_exception",
            "reason": "Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
            "caused_by": {
              "type": "s_s_l_handshake_exception",
              "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
              "caused_by": {
                "type": "validator_exception",
                "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
                "caused_by": {
                  "type": "sun_cert_path_builder_exception",
                  "reason": "unable to find valid certification path to requested target"
                }
              }
            }
          }
        }
      },
      "status": 500
    },
    "statusCode": 500,
    "headers": {
      "x-opaque-id": "3a8feac6-c37b-4d43-a0a2-421bd98b8c84;kibana:application:management:",
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json;charset=utf-8",
      "content-length": "1167"
    },
    "meta": {
      "context": null,
      "request": {
        "params": {
          "method": "POST",
          "path": "/_snapshot/IST_ELASTIC_BKP/_verify",
          "querystring": "",
          "headers": {
            "user-agent": "Kibana/8.6.2",
            "x-elastic-product-origin": "kibana",
            "authorization": "Basic ZWxhc3RpYzpGWXFVVDIySkJEaTlIQ3pZUkEqNw==",
            "x-opaque-id": "3a8feac6-c37b-4d43-a0a2-421bd98b8c84;kibana:application:management:",
            "x-elastic-client-meta": "es=8.4.0p,js=16.18.1,t=8.2.0,hc=16.18.1",
            "accept": "application/vnd.elasticsearch+json; compatible-with=8,text/plain"
          }
        },
        "options": {
          "opaqueId": "3a8feac6-c37b-4d43-a0a2-421bd98b8c84;kibana:application:management:",
          "headers": {
            "x-elastic-product-origin": "kibana",
            "user-agent": "Kibana/8.6.2",
            "authorization": "Basic ZWxhc3RpYzpGWXFVVDIySkJEaTlIQ3pZUkEqNw==",
            "x-opaque-id": "3a8feac6-c37b-4d43-a0a2-421bd98b8c84",
            "x-elastic-client-meta": "es=8.4.0p,js=16.18.1,t=8.2.0,hc=16.18.1"
          }
        },
        "id": 1
      },
      "name": "elasticsearch-js",
      "connection": {
        "url": "https://10.107.94.218:9090/",
        "id": "https://10.107.94.218:9090/",
        "headers": {},
        "status": "alive"
      },
      "attempts": 0,
      "aborted": false
    },
    "warnings": null
  }
}
4

Sounds like it's related to certificates. I don't have enough knowledge sadly to help on that but may be if you describe exactly how you are running Elasticsearch (which exact version and distribution) and how you installed it, that could help others to add more ideas?

I'd try to create a cluster on cloud.elastic.co and test if you can add your S3 repo. If so, that'd mean that something is wrong with the way you installed Elasticsearch. Could be the java version, the certificate....

5

I am using basic license and using Elasticsearch 8.6.2 in a cluster mode

I have 2 nodes in my cluster. I have also tried to add crt file of my s3 bucket endpoint in elastic jdk keytool but still no luck.

6

@leandrojmp , @stephenb can you guys help on this thread?

7

Can anyone help me on this topic?

8

What do you mean? Did you get an error message?
What exact command did you run?

I saw this in the documentation (emphasis is mine):

This SunCertPathBuilderException indicates that a certificate was returned during the handshake that is not trusted. This message is seen on the client side of the connection. The SSLException is seen on the server side of the connection. The CA certificate that signed the returned certificate was not found in the keystore or truststore and needs to be added to trust this certificate.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.