Snapshot to minio S3 error

Elastic Stack Elasticsearch
1

Hi,

Elasticsearch 7.10.
I'm trying to add an S3 snapshot repository, the endpoint is Minio.

The repository-s3 plugin is installed on all nodes, and restarted.
I've set the keys in the keystore:

elasticsearch-keystore add s3.client.default.access_key
elasticsearch-keystore add s3.client.default.secret_key

Triggered a POST _nodes/reload_secure_settings

Then I add the snapshot repository:

PUT _snapshot/minio_s3
{
  "type": "s3",
  "settings": {
    "bucket": "elastic-engineering",
    "client": "default",
    "endpoint": "elastic-s3:9000",
    "path_style_access": "true"
  }
}

Which returns this error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[minio_s3] path  is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[minio_s3] path  is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object [tests-QQdH9nWXSkuOhKC4zkE2jQ/master.dat] using a single upload",
      "caused_by" : {
        "type" : "sdk_client_exception",
        "reason" : "sdk_client_exception: Failed to connect to service endpoint: ",
        "caused_by" : {
          "type" : "i_o_exception",
          "reason" : "Connect timed out"
        }
      }
    }
  },
  "status" : 500
}

I verified, all nodes are able to connect to https://elastic-s3:9000 using curl.

The elasticsearch log shows:

[INFO ][o.e.r.RepositoriesService] [elastic014] update repository [minio_s3]
[WARN ][o.e.r.s.S3BlobStore      ] [elastic014] Expected request count to be tracked for request [PUT https://elastic-s3:9000 /elastic-engineering/tests-QQdH9nWXSkuOhKC4zkE2jQ/master.dat Headers: (Content-Length: 22, Content-Type: application/octet-stream, x-amz-acl: private, x-amz-storage-class: STANDARD, ) ] but found not count.
2

Can you try again with ?error_trace:

PUT _snapshot/minio_s3?error_trace
{
  "type": "s3",
  "settings": {
    "bucket": "elastic-engineering",
    "client": "default",
    "endpoint": "elastic-s3:9000",
    "path_style_access": "true"
  }
}

Connect timed out does rather suggest it can't connect to the endpoint. Another simple way to dig deeper is to take packet captures and see to what it's failing to connect, and why.

3

Am I sending the endpoint for the S3 repository the right way?
It looks like something is not getting through, cause I see the message: [sdk_client_exception: Failed to connect to service endpoint: ]; nested: IOException[Connect timed out];

The endpoint is empty in there?
When I do a tcpdump during this request, I only see traffic to the link local IP:

IP (tos 0x0, ttl 64, id 40412, offset 0, flags [DF], proto TCP (6), length 60)
    77.x.x.x.35600 > 169.254.169.254.80: Flags [S], cksum 0x0051 (incorrect -> 0xec1d), seq 410148014, win 64240, options [mss 1460,sackOK,TS val 89610999 ecr 0,nop,wscale 7], length 0
IP (tos 0x0, ttl 64, id 5234, offset 0, flags [DF], proto TCP (6), length 60)
    77.x.x.x.35602 > 169.254.169.254.80: Flags [S], cksum 0x0051 (incorrect -> 0x7814), seq 3703638141, win 64240, options [mss 1460,sackOK,TS val 89612000 ecr 0,nop,wscale 7], length 0
4

That's the EC2 metadata endpoint, which means that this node doesn't have its own credentials. Are you sure that s3.client.default.access_key and s3.client.default.secret_key are set on every node?

5

It was not clear to me that these secure keys needed to be set on each node.
I've read the secure settings manual now, and it's mentioned there this should be done on all nodes indeed.

Works fine after doing that.

Sorry for not rtfm and thanks!

2 Likes
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.