# S3 - setting and verifying repo issue
**URL:** https://discuss.elastic.co/t/s3-setting-and-verifying-repo-issue/286661
**Category:** Elasticsearch
**Tags:** snapshot-and-restore
**Created:** [October 13, 2021, 8:35pm UTC](https://discuss.elastic.co/t/s3-setting-and-verifying-repo-issue/286661 "2021-10-13T20:35:42Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@vee](https://discuss.elastic.co/u/vee)
#### Post date: [October 13, 2021, 8:35pm UTC](https://discuss.elastic.co/t/s3-setting-and-verifying-repo-issue/286661/1 "2021-10-13T20:35:42Z")
Trying our in-house S3 compatible storage device as frozen tier, but seeing errors when verifying the repository. Got a confirmation from elastic support that the one we have is fully supported by S3 protocol.
Here are the things done so far:
Elastic v7.13 on RHEL
1. Made sure no n/w issue between elastic and the in-house S3 endpoint (residing in same DC)
2. Added the below config to Elasticsearch.yml:
```auto
s3.client.es_s3.endpoint: "host:port"
s3.client.es_s3.protocol: https
```
1. Added access and secret keys to Elasticsearch keystore:
```auto
$bin/elasticsearch-keystore list
s3.client.es_s3.access_key
s3.client.es_s3.secret_key
```
1. Enabled the aws s3 plugin
2. Since it's https URL, added it's cert to default jdk/lib/security/cacerts truststore, and also to both Elasticsearch keystores (transport & http).
3. openssl connect using the crt file works fine
4. AWS client connection works fine when using the crt file and also when I add crt to a pem keystore.
```auto
eg: /usr/local/bin/aws s3 ls --endpoint-url=https://host:port s3://bucket1/ --ca-bundle $AWS_CA_BUNDLE
```
1. Where exactly should the client cert be added to for setting up the repo? It seems it's not able to utilize all the keystores/truststores available in Elastic install/config paths. Tried changing the "xpack.security.http.ssl.keystore.path" to point to pem format file instead of p12 but elastic wouldn't even start when I do so.
Here's the error when I try to run the PUT snapshot command:
```auto
# PUT /_snapshot/es_s3
{
"error" : {
"root_cause" : [
{
"type" : "repository_verification_exception",
"reason" : "[es_s3] path is not accessible on master node"
}
],
"type" : "repository_verification_exception",
"reason" : "[es_s3] path is not accessible on master node",
"caused_by" : {
"type" : "i_o_exception",
"reason" : "Unable to upload object [tests-CardaXhOS_KsrWSs-pSKcg/master.dat] using a single upload",
"caused_by" : {
"type" : "sdk_client_exception",
"reason" : "Failed to connect to service endpoint: ",
"caused_by" : {
"type" : "socket_timeout_exception",
"reason" : "Read timed out"
}
}
}
},
"status" : 500
}
```
If you have any suggestions, please let me know. Thanks for your time.
---
### Author:  [@zx8086](https://discuss.elastic.co/u/zx8086)
#### Post date: [October 13, 2021, 9:12pm UTC](https://discuss.elastic.co/t/s3-setting-and-verifying-repo-issue/286661/2 "2021-10-13T21:12:58Z")
What is your full PUT command for the S3 repository like my example
```auto
PUT _snapshot/my_s3_repository
{
"type": "s3",
"settings": {
"bucket": "elasticsearch.mydomain.com-snapshots-eu-central-1"
}
}
```
or the result from
```auto
GET /_snapshot/_all
```
---
### Author:  [@vee](https://discuss.elastic.co/u/vee)
#### Post date: [October 13, 2021, 9:29pm UTC](https://discuss.elastic.co/t/s3-setting-and-verifying-repo-issue/286661/3 "2021-10-13T21:29:49Z")
Here it is:
```auto
# with path style added
PUT /_snapshot/elastic_search_s3
{
"type": "s3",
"settings": {
"bucket": "bucket1",
"endpoint": "host:port",
"protocol": "https",
"path_style_access": "true"
}
}
```
---
### Author:  [@zx8086](https://discuss.elastic.co/u/zx8086)
#### Post date: [October 13, 2021, 9:39pm UTC](https://discuss.elastic.co/t/s3-setting-and-verifying-repo-issue/286661/4 "2021-10-13T21:39:54Z")
> [@vee](#):
>
> ```auto
> "endpoint": "host:port",
> "protocol": "https",
> "path_style_access": "true"
>
> ```
What happens without these settings ? Just to rule out SSL issues.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [October 14, 2021, 2:21am UTC](https://discuss.elastic.co/t/s3-setting-and-verifying-repo-issue/286661/5 "2021-10-14T02:21:47Z")
> [@vee](#):
>
> ```auto
> $bin/elasticsearch-keystore list
> s3.client.es_s3.access_key
> s3.client.es_s3.secret_key
>
> ```
I think your secrets should be
```auto
s3.client.default.access_key
s3.client.default.secret_key
```
Unless you set the `client` name to `es_s3` in the repository settings? Did you?
> `client`
>
> The name of the [S3 client](https://www.elastic.co/guide/en/elasticsearch/plugins/7.15/repository-s3-client.html) to use to connect to S3. Defaults to `default`.
> [@vee](#):
>
> `# PUT /_snapshot/es_s3`
It is not the name of the snapshot
---
### Author:  [@vee](https://discuss.elastic.co/u/vee)
#### Post date: [October 14, 2021, 2:48pm UTC](https://discuss.elastic.co/t/s3-setting-and-verifying-repo-issue/286661/6 "2021-10-14T14:48:59Z")
I have the client name set to es\_s3 (same as repo name) and so the secrets are named as such: s3.client.es\_s3.access\_key & s3.client.es\_s3.secret\_key.
Good point on trying with http protocol, I tested against it this morning and still notice the same issue.
Changed the Elasticsearch.yml settings to:
```auto
s3.client.es_s3.endpoint: "host:httpport"
s3.client.es_s3.protocol: http
```
Plugin has been enabled:
```auto
bin/elasticsearch-plugin list:
repository-s3
```
Dev tools command:
```auto
DELETE _snapshot/es_s3
PUT /_snapshot/es_s3
{
"type": "s3",
"settings": {
"bucket": "bucket1",
"endpoint": "host:httpport",
"protocol": "http"
}
}
```
Error msg:
```auto
[2021-10-14T09:41:52,945][WARN][r.suppressed] [ingest1] path: /_snapshot/es_s3, params: {pretty=true, repository=es_s3}
org.elasticsearch.repositories.RepositoryVerificationException: [es_s3] path is not accessible on master node
Caused by: java.io.IOException: Unable to upload object [tests-WarEYldYSLyB-kW-08AebQ/master.dat] using a single upload
.....
Caused by: com.amazonaws.SdkClientException: Failed to connect to service endpoint:
.....
Caused by: java.net.SocketTimeoutException: Read timed out
```
Not sure what else might be missing!!
---
### Author:  [@vee](https://discuss.elastic.co/u/vee)
#### Post date: [October 14, 2021, 3:31pm UTC](https://discuss.elastic.co/t/s3-setting-and-verifying-repo-issue/286661/7 "2021-10-14T15:31:35Z")
I tried NOT to give a client name and instead used the default (with minimal expected configuration). Added the default access& secret keys to elastic keystore - however see this error when attempted to do the same:
Elasticsearch.yml:
```auto
s3.client.default.endpoint: "host:httpport"
s3.client.default.protocol: http
```
verified keystore list:
```auto
bin/elasticsearch-keystore list
s3.client.default.access_key
s3.client.default.secret_key
```
Devtools commands:
```auto
DELETE _snapshot/es_s3
PUT /_snapshot/es_s3
{
"type": "s3",
"settings": {
"bucket": "bucket1"
}
}
```
Error msg:
```auto
org.elasticsearch.repositories.RepositoryVerificationException: [es_s3] path is not accessible on master node
Caused by: java.io.IOException: Unable to upload object [tests-bsxr8NkARKewFF0NPaoFwg/master.dat] using a single upload
...
Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: bucket1.host
...
Caused by: java.net.UnknownHostException: bucket1.host
```
verified telnet from elastic to host on httpport works fine - so connectivity issue:
---
### Author:  [@zx8086](https://discuss.elastic.co/u/zx8086)
#### Post date: [October 14, 2021, 3:34pm UTC](https://discuss.elastic.co/t/s3-setting-and-verifying-repo-issue/286661/8 "2021-10-14T15:34:52Z")
@vee
What happens when you just use the basic configuration as documented without the additional parameters, is this possible ?
> **[Client Settings | Elasticsearch Plugins and Integrations \[7.15\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/plugins/7.15/repository-s3-client.html)**
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [October 14, 2021, 3:42pm UTC](https://discuss.elastic.co/t/s3-setting-and-verifying-repo-issue/286661/9 "2021-10-14T15:42:34Z")
How Many Nodes? What Version?
You can turn up the logs to DEBUG and see if you get more verbose error logs.
> [@vee](#):
>
> ```auto
> Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: bucket1.host
> ...
> Caused by: java.net.UnknownHostException: bucket1.host