# Snapshot to minio S3 error **URL:** \ **Category:** Elasticsearch\ **Created:** [January 6, 2021, 4:06pm UTC](https://discuss.elastic.co/t/snapshot-to-minio-s3-error/260365 "2021-01-06T16:06:26Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![gregorys](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gregorys/32/67574_2.png) [@gregorys](https://discuss.elastic.co/u/gregorys)\ **Post date:** [January 6, 2021, 4:06pm UTC](https://discuss.elastic.co/t/snapshot-to-minio-s3-error/260365/1 "2021-01-06T16:06:26Z")
Hi, Elasticsearch 7.10. I'm trying to add an S3 snapshot repository, the endpoint is Minio. The repository-s3 plugin is installed on all nodes, and restarted. I've set the keys in the keystore: ```auto elasticsearch-keystore add s3.client.default.access_key elasticsearch-keystore add s3.client.default.secret_key ``` Triggered a `POST _nodes/reload_secure_settings` Then I add the snapshot repository: ```auto PUT _snapshot/minio_s3 { "type": "s3", "settings": { "bucket": "elastic-engineering", "client": "default", "endpoint": "elastic-s3:9000", "path_style_access": "true" } } ``` Which returns this error: ```auto { "error" : { "root_cause" : [ { "type" : "repository_verification_exception", "reason" : "[minio_s3] path is not accessible on master node" } ], "type" : "repository_verification_exception", "reason" : "[minio_s3] path is not accessible on master node", "caused_by" : { "type" : "i_o_exception", "reason" : "Unable to upload object [tests-QQdH9nWXSkuOhKC4zkE2jQ/master.dat] using a single upload", "caused_by" : { "type" : "sdk_client_exception", "reason" : "sdk_client_exception: Failed to connect to service endpoint: ", "caused_by" : { "type" : "i_o_exception", "reason" : "Connect timed out" } } } }, "status" : 500 } ``` I verified, all nodes are able to connect to [https://elastic-s3:9000](https://elastic-s3:9000) using curl. The elasticsearch log shows: ```auto [INFO][o.e.r.RepositoriesService] [elastic014] update repository [minio_s3] [WARN][o.e.r.s.S3BlobStore] [elastic014] Expected request count to be tracked for request [PUT https://elastic-s3:9000 /elastic-engineering/tests-QQdH9nWXSkuOhKC4zkE2jQ/master.dat Headers: (Content-Length: 22, Content-Type: application/octet-stream, x-amz-acl: private, x-amz-storage-class: STANDARD, )] but found not count. ``` ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 6, 2021, 6:19pm UTC](https://discuss.elastic.co/t/snapshot-to-minio-s3-error/260365/2 "2021-01-06T18:19:11Z")
Can you try again with `?error_trace`: ```auto PUT _snapshot/minio_s3?error_trace { "type": "s3", "settings": { "bucket": "elastic-engineering", "client": "default", "endpoint": "elastic-s3:9000", "path_style_access": "true" } } ``` `Connect timed out` does rather suggest it can't connect to the endpoint. Another simple way to dig deeper is to take packet captures and see to what it's failing to connect, and why. ---
**Author:** ![gregorys](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gregorys/32/67574_2.png) [@gregorys](https://discuss.elastic.co/u/gregorys)\ **Post date:** [January 6, 2021, 7:45pm UTC](https://discuss.elastic.co/t/snapshot-to-minio-s3-error/260365/3 "2021-01-06T19:45:41Z")
Am I sending the endpoint for the S3 repository the right way? It looks like something is not getting through, cause I see the message: `[sdk_client_exception: Failed to connect to service endpoint:]; nested: IOException[Connect timed out];` The endpoint is empty in there? When I do a tcpdump during this request, I only see traffic to the link local IP: ```auto IP (tos 0x0, ttl 64, id 40412, offset 0, flags [DF], proto TCP (6), length 60) 77.x.x.x.35600 > 169.254.169.254.80: Flags [S], cksum 0x0051 (incorrect -> 0xec1d), seq 410148014, win 64240, options [mss 1460,sackOK,TS val 89610999 ecr 0,nop,wscale 7], length 0 IP (tos 0x0, ttl 64, id 5234, offset 0, flags [DF], proto TCP (6), length 60) 77.x.x.x.35602 > 169.254.169.254.80: Flags [S], cksum 0x0051 (incorrect -> 0x7814), seq 3703638141, win 64240, options [mss 1460,sackOK,TS val 89612000 ecr 0,nop,wscale 7], length 0 ``` ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 6, 2021, 7:55pm UTC](https://discuss.elastic.co/t/snapshot-to-minio-s3-error/260365/4 "2021-01-06T19:55:32Z")
> [@gregorys](#): > > `169.254.169.254.80` That's the EC2 metadata endpoint, which means that this node doesn't have its own credentials. Are you sure that `s3.client.default.access_key` and `s3.client.default.secret_key` are set on _every_ node? ---
**Author:** ![gregorys](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gregorys/32/67574_2.png) [@gregorys](https://discuss.elastic.co/u/gregorys)\ **Post date:** [January 6, 2021, 8:04pm UTC](https://discuss.elastic.co/t/snapshot-to-minio-s3-error/260365/5 "2021-01-06T20:04:37Z")
It was not clear to me that these secure keys needed to be set on each node. I've read the secure settings manual now, and it's mentioned there this should be done on all nodes indeed. Works fine after doing that. Sorry for not rtfm and thanks! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 3, 2021, 8:04pm UTC](https://discuss.elastic.co/t/snapshot-to-minio-s3-error/260365/6 "2021-02-03T20:04:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.