# Snapshot repository-s3 registry failed in 7.17

**URL:** <https://discuss.elastic.co/t/snapshot-repository-s3-registry-failed-in-7-17/360422>\
**Category:** Elasticsearch\
**Tags:** docker, snapshot-and-restore\
**Created:** [May 29, 2024, 5:00am UTC](https://discuss.elastic.co/t/snapshot-repository-s3-registry-failed-in-7-17/360422 "2024-05-29T05:00:54Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![dove-young](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dove-young/32/133625_2.png) [@dove-young](https://discuss.elastic.co/u/dove-young)\
**Post date:** [May 29, 2024, 5:00am UTC](https://discuss.elastic.co/t/snapshot-repository-s3-registry-failed-in-7-17/360422/1 "2024-05-29T05:00:54Z")

</div>

I run elasticsearch docker container as described in quickstart [Quick start | Elasticsearch Guide [7.17] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/getting-started.html),

then I installed `repository-s3` plugin and restarted container.

I have a local minio server act as S3 storage. But I am not able to registry a S3 repository in Elasticsearch. The error msg said `"reason" : "Failed to connect to service endpoint: "`, but it does not indicate what `service endpoint` it connected to.

```auto
[root@st2-tunnel-dev1]# docker exec -it es01-test /bin/bash
root@03caafc38d92:/usr/share/elasticsearch# bin/elasticsearch-keystore add s3.client.default.access_key
Enter value for s3.client.default.access_key:
root@03caafc38d92:/usr/share/elasticsearch# bin/elasticsearch-keystore add s3.client.default.secret_key
Enter value for s3.client.default.secret_key:

root@03caafc38d92:/usr/share/elasticsearch# curl -X PUT "http://localhost:9200/_snapshot/my_s3_repo?pretty" -H 'Content-Type: application/json' -d'
> {
> "type": "s3",
> "settings": {
> "endpoint": "http://<my-minio-ip>:9000",
> "protocol": "http",
> "region": "eu-central-1",
> "bucket": "elastic-o1-726938",
> "base_path": "/usr/share/elasticsearch/data/nodes/0",
> "client": "default",
> "path_style_access": "true"
> }
> }'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_s3_repo] path [/usr/share/elasticsearch/data/nodes/0] is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[my_s3_repo] path [/usr/share/elasticsearch/data/nodes/0] is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object [/usr/share/elasticsearch/data/nodes/0/tests-e8hrHXQyQiOIgHh0sypi5w/master.dat] using a single upload",
      "caused_by" : {
        "type" : "sdk_client_exception",
        "reason" : "Failed to connect to service endpoint: ",
        "caused_by" : {
          "type" : "socket_timeout_exception",
          "reason" : "Connect timed out"
        }
      }
    }
  },
  "status" : 500
}

```

I also tried to adjust endpoint format, then I got error like this

```auto
root@03caafc38d92:/usr/share/elasticsearch# curl -X PUT "http://localhost:9200/_snapshot/my_s3_repo?pretty" -H 'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
    "endpoint": "<my-minio-ip>:9000",
    "protocol": "http",
    "region": "eu-central-1",
    "bucket": "elastic-o1-726938",
    "base_path": "/usr/share/elasticsearch/data/nodes/0",
    "client": "default",
    "path_style_access": "true"
  }
}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_exception",
        "reason" : "[my_s3_repo] Could not determine repository generation from root blobs"
      }
    ],
    "type" : "repository_exception",
    "reason" : "[my_s3_repo] Could not determine repository generation from root blobs",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Exception when listing blobs by prefix [index-]",
      "caused_by" : {
        "type" : "sdk_client_exception",
        "reason" : "Failed to connect to service endpoint: ",
        "caused_by" : {
          "type" : "socket_timeout_exception",
          "reason" : "Connect timed out"
        }
      }
    }
  },
  "status" : 500
}

```

but I can access to my minio server via `curl` in this container.

```auto
root@03caafc38d92:/usr/share/elasticsearch# curl -k -H 'Host: <my-minio-ip>:9000' -H 'Date: Wed, 29 May 2024 04:58:50 +0000' -H 'Content-Type: application/zstd' -H 'Authorization: AWS minioadmin:18Hr3bux0qN+JM6wxWiKdctEc0k=' http://<my-minio-ip>:9000/
<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4</ID><DisplayName>minio</DisplayName></Owner><Buckets><Bucket><Name>elastic-o1-726938</Name><CreationDate>2024-04-18T05:48:23.891Z</CreationDate></Bucket><Bucket><Name>my-bucket-longterm</Name><CreationDate>2024-04-26T03:03:08.535Z</CreationDate></Bucket><Bucket><Name>mybucket</Name><CreationDate>2024-04-11T10:22:01.282Z</CreationDate></Bucket><Bucket><Name>oadp-o1-726938</Name><CreationDate>2024-04-12T08:39:08.429Z</CreationDate></Bucket><Bucket><Name>rawspanbucket</Name><CreationDate>2024-05-08T05:09:58.909Z</CreationDate></Bucket><Bucket><Name>test-bucket</Name><CreationDate>2024-04-23T04:33:19.262Z</CreationDate></Bucket><Bucket><Name>test-store</Name><CreationDate>2024-04-03T01:27:13.698Z</CreationDate></Bucket><Bucket><Name>veleco-hgliu</Name><CreationDate>2024-04-12T01:54:10.483Z</CreationDate></Bucket><Bucket><Name>veleco-o1-726938</Name><CreationDate>2024-04-12T01:50:04.447Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>root@03caafc38d92:/usr/share/elasticsearch#
root@03caafc38d92:/usr/share/elasticsearch#

```

if I remove `base_path` from payload, I got similar error `path is not accessible on master node` and it also not indicate what `path ` is.

```auto
root@03caafc38d92:/usr/share/elasticsearch# curl -X PUT "http://localhost:9200/_snapshot/my_s3_repo2?pretty" -H 'Content-Type: application/json' -d'
> {
> "type": "s3",
> "settings": {
> "endpoint": "9.112.252.135:9000",
> "protocol": "http",
> "region": "eu-central-1",
> "bucket": "elastic-o1-726938",
> "client": "default",
> "path_style_access": "true"
> }
> }'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_s3_repo2] path is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[my_s3_repo2] path is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object [tests-GacOt_JST22bgxGuIZuCBQ/master.dat] using a single upload",
      "caused_by" : {
        "type" : "sdk_client_exception",
        "reason" : "Failed to connect to service endpoint: ",
        "caused_by" : {
          "type" : "socket_timeout_exception",
          "reason" : "Connect timed out"
        }
      }
    }
  },
  "status" : 500
}

```

---

<div class="post-metadata">

**Author:** ![dove-young](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dove-young/32/133625_2.png) [@dove-young](https://discuss.elastic.co/u/dove-young)\
**Post date:** [May 29, 2024, 6:07am UTC](https://discuss.elastic.co/t/snapshot-repository-s3-registry-failed-in-7-17/360422/2 "2024-05-29T06:07:53Z")

</div>

finally I got it. the `base_path` must in the `path.repo` setting in the `elasticsearch.yml`

```auto
root@03caafc38d92:/usr/share/elasticsearch# curl -X PUT "http://localhost:9200/_snapshot/my_s3_repo3?pretty" -H 'Content-Type: application/json' -d'
> {
> "type": "s3",
> "settings": {
> "endpoint": "<my-minio-ip>:9000",
> "protocol": "http",
> "region": "eu-central-1",
> "bucket": "elastic-o1-726938",
> "client": "default",
> "base_path": "/tmp/backup",
> "path_style_access": "true"
> }
> }'
{
  "acknowledged" : true
}
root@03caafc38d92:/usr/share/elasticsearch# cat config/elasticsearch.yml
cluster.name: "docker-cluster"
network.host: 0.0.0.0
path.repo: ["/tmp/backup"]
root@03caafc38d92:/usr/share/elasticsearch#

```

---

<div class="post-metadata">

**Author:** ![dove-young](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dove-young/32/133625_2.png) [@dove-young](https://discuss.elastic.co/u/dove-young)\
**Post date:** [May 29, 2024, 6:21am UTC](https://discuss.elastic.co/t/snapshot-repository-s3-registry-failed-in-7-17/360422/3 "2024-05-29T06:21:20Z")

</div>

But I still not able to register a S3 repository on elasticsearch running in openshift/Kubernetes

```auto
[root@st2-tunnel-dev1]# oc rsh -n instana-elastic instana-es-default-0
Defaulted container "elasticsearch" out of: elasticsearch, elastic-internal-init-filesystem (init), elastic-internal-suspend (init)
sh-5.1$ ELASTIC_PASSWORD=66q45Kg64qd03fbCiVm7y9wj
sh-5.1$ bin/elasticsearch-keystore add s3.client.default.access_key
warning: usage of JAVA_HOME is deprecated, use ES_JAVA_HOME
Enter value for s3.client.default.access_key:
sh-5.1$ bin/elasticsearch-keystore add s3.client.default.secret_key
warning: usage of JAVA_HOME is deprecated, use ES_JAVA_HOME
Enter value for s3.client.default.secret_key:
sh-5.1$
sh-5.1$ curl -X PUT "http://instana-es-http:9200/_snapshot/my_s3_repo?pretty" -H 'Content-Type: application/json' -u elastic:$ELASTIC_PASSWORD -d'
{
  "type": "s3",
  "settings": {
    "endpoint": "<my-minio-ip>:9000",
    "protocol": "http",
    "region": "eu-central-1",
    "bucket": "elastic-o1-726938",
    "base_path": "/tmp",
    "client": "default",
    "path_style_access": "true"
  }
}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_s3_repo] path [/tmp] is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[my_s3_repo] path [/tmp] is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object [/tmp/tests-ajpHGZF5SciLw6Y1XwNP4g/master.dat] using a single upload",
      "caused_by" : {
        "type" : "sdk_client_exception",
        "reason" : "sdk_client_exception: Failed to connect to service endpoint: ",
        "caused_by" : {
          "type" : "i_o_exception",
          "reason" : "Connection refused"
        }
      }
    }
  },
  "status" : 500
}

```

here is what in `config/elasticsearch.yml`

```auto
path:
    data: /usr/share/elasticsearch/data
    logs: /usr/share/elasticsearch/logs
    repo:
        - /tmp

```

here is what I defined in elasticsearch CR

```auto
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: instana
spec:
  version: 7.17.15
  nodeSets:
    - name: default
      count: 3
      config:
        node.master: true
        node.data: true
        node.ingest: true
        node.store.allow_mmap: false
        path.repo: ["/tmp"]

```

---

<div class="post-metadata">

**Author:** ![dove-young](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dove-young/32/133625_2.png) [@dove-young](https://discuss.elastic.co/u/dove-young)\
**Post date:** [May 29, 2024, 6:31am UTC](https://discuss.elastic.co/t/snapshot-repository-s3-registry-failed-in-7-17/360422/4 "2024-05-29T06:31:25Z")

</div>

I cannot even create a `fs` repository in elastic running in OCP/K8S

```auto
sh-5.1$ curl -X PUT 'http://instana-es-http:9200/_snapshot/my_fs_2?pretty' -H 'Content-Type: application/json' -u elastic:$ELASTIC_PASSWORD -d '
{
  "type": "fs",
  "settings": {
    "location": "/tmp"
  }
}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_fs_2] [[a-7DjAEeQiWQF5TplyJPyw, 'RemoteTransportException[[instana-es-default-2][10.254.32.66:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_fs_2] a file written by master to the store [/tmp] cannot be accessed on the node [{instana-es-default-2}{a-7DjAEeQiWQF5TplyJPyw}{zbN9rtONQJKogBbnbxzKTA}{10.254.32.66}{10.254.32.66:9300}{cdfhilmrstw}{k8s_node_name=worker3.o1-743345.cp.fyre.ibm.com, ml.machine_memory=2147483648, xpack.installed=true, transform.node=true, ml.max_open_jobs=512, ml.max_jvm_size=1073741824}]. This might indicate that the store [/tmp] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node]; nested: NoSuchFileException[/tmp/tests-XEAkE8J_Sr-b54dzpD-g0Q/master.dat];'], [Waf822duS4ufvW7ryX2PRA, 'RemoteTransportException[[instana-es-default-1][10.254.36.72:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_fs_2] a file written by master to the store [/tmp] cannot be accessed on the node [{instana-es-default-1}{Waf822duS4ufvW7ryX2PRA}{MlJ4UVthSgO_bN4yHGmzDQ}{10.254.36.72}{10.254.36.72:9300}{cdfhilmrstw}{k8s_node_name=worker5.o1-743345.cp.fyre.ibm.com, ml.machine_memory=2147483648, xpack.installed=true, transform.node=true, ml.max_open_jobs=512, ml.max_jvm_size=1073741824}]. This might indicate that the store [/tmp] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node]; nested: NoSuchFileException[/tmp/tests-XEAkE8J_Sr-b54dzpD-g0Q/master.dat];']]"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[my_fs_2] [[a-7DjAEeQiWQF5TplyJPyw, 'RemoteTransportException[[instana-es-default-2][10.254.32.66:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_fs_2] a file written by master to the store [/tmp] cannot be accessed on the node [{instana-es-default-2}{a-7DjAEeQiWQF5TplyJPyw}{zbN9rtONQJKogBbnbxzKTA}{10.254.32.66}{10.254.32.66:9300}{cdfhilmrstw}{k8s_node_name=worker3.o1-743345.cp.fyre.ibm.com, ml.machine_memory=2147483648, xpack.installed=true, transform.node=true, ml.max_open_jobs=512, ml.max_jvm_size=1073741824}]. This might indicate that the store [/tmp] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node]; nested: NoSuchFileException[/tmp/tests-XEAkE8J_Sr-b54dzpD-g0Q/master.dat];'], [Waf822duS4ufvW7ryX2PRA, 'RemoteTransportException[[instana-es-default-1][10.254.36.72:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_fs_2] a file written by master to the store [/tmp] cannot be accessed on the node [{instana-es-default-1}{Waf822duS4ufvW7ryX2PRA}{MlJ4UVthSgO_bN4yHGmzDQ}{10.254.36.72}{10.254.36.72:9300}{cdfhilmrstw}{k8s_node_name=worker5.o1-743345.cp.fyre.ibm.com, ml.machine_memory=2147483648, xpack.installed=true, transform.node=true, ml.max_open_jobs=512, ml.max_jvm_size=1073741824}]. This might indicate that the store [/tmp] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node]; nested: NoSuchFileException[/tmp/tests-XEAkE8J_Sr-b54dzpD-g0Q/master.dat];']]"
  },
  "status" : 500
}
sh-5.1$

```

---

<div class="post-metadata">

**Author:** ![dove-young](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dove-young/32/133625_2.png) [@dove-young](https://discuss.elastic.co/u/dove-young)\
**Post date:** [May 29, 2024, 6:41am UTC](https://discuss.elastic.co/t/snapshot-repository-s3-registry-failed-in-7-17/360422/5 "2024-05-29T06:41:36Z")

</div>

When I scale down to 1 instace, I can define `fs` type repository in K8S

```auto
[root@st2-tunnel-dev1]# oc rsh -n instana-elastic instana-es-default-0
Defaulted container "elasticsearch" out of: elasticsearch, elastic-internal-init-filesystem (init), elastic-internal-suspend (init)
sh-5.1$ ELASTIC_PASSWORD=66q45Kg64qd03fbCiVm7y9wj
sh-5.1$ curl -X PUT 'http://instana-es-http:9200/_snapshot/my_fs_2?pretty' -H 'Content-Type: application/json' -u elastic:$ELASTIC_PASSWORD -d '
{
  "type": "fs",
  "settings": {
    "location": "/tmp"
  }
}'
{
  "acknowledged" : true
}
sh-5.1$

```

But I still not able to define `s3` type registry when there is 1 elastic instance

```auto
sh-5.1$ curl -X PUT "http://instana-es-http:9200/_snapshot/my_s3_repo3?pretty" -H 'Content-Type: application/json' -u elastic:$ELASTIC_PASSWORD -d'
{
  "type": "s3",
  "settings": {
    "endpoint": "9.112.252.135:9000",
    "protocol": "http",
    "region": "eu-central-1",
    "bucket": "elastic-o1-726938",
    "base_path": "/tmp",
    "client": "default",
    "path_style_access": "true"
  }
}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_s3_repo3] path [/tmp] is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[my_s3_repo3] path [/tmp] is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object [/tmp/tests-7vIME0laTKufgaxdhlI2PQ/master.dat] using a single upload",
      "caused_by" : {
        "type" : "sdk_client_exception",
        "reason" : "Failed to connect to service endpoint: ",
        "caused_by" : {
          "type" : "connect_exception",
          "reason" : "Connection refused"
        }
      }
    }
  },
  "status" : 500
}
sh-5.1$

```

when I retry, it output another error mssage

```auto
sh-5.1$ curl -X PUT "http://instana-es-http:9200/_snapshot/my_s3_repo3?pretty" -H 'Content-Type: application/json' -u elastic:$ELASTIC_PASSWORD -d'
{
  "type": "s3",
  "settings": {
    "endpoint": "9.112.252.135:9000",
    "protocol": "http",
    "region": "eu-central-1",
    "bucket": "elastic-o1-726938",
    "base_path": "/tmp",
    "client": "default",
    "path_style_access": "true"
  }
}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_exception",
        "reason" : "[my_s3_repo3] Could not determine repository generation from root blobs"
      }
    ],
    "type" : "repository_exception",
    "reason" : "[my_s3_repo3] Could not determine repository generation from root blobs",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Exception when listing blobs by prefix [index-]",
      "caused_by" : {
        "type" : "sdk_client_exception",
        "reason" : "Failed to connect to service endpoint: ",
        "caused_by" : {
          "type" : "connect_exception",
          "reason" : "Connection refused"
        }
      }
    }
  },
  "status" : 500
}
sh-5.1$

```

---

<div class="post-metadata">

**Author:** ![dove-young](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dove-young/32/133625_2.png) [@dove-young](https://discuss.elastic.co/u/dove-young)\
**Post date:** [May 30, 2024, 3:19am UTC](https://discuss.elastic.co/t/snapshot-repository-s3-registry-failed-in-7-17/360422/6 "2024-05-30T03:19:08Z")

</div>

so here are the questions for elasticsearch 7.17.x installed in K8S/OCP via helm operator version=2.9.0 from elastic [https://helm.elastic.co](https://helm.elastic.co):

1. Is it true that even in `s3` repository registry payload, the `base_path` also need to be a shared file system between all elastic pods?
2. How can I add an additional volume to elasticsearch pod in Elasticsearch CR? I searched Elasticsearch CRD, I don't find any `volumeMount` item in the CRD. I can add a new volume via `volumeClaimTemplate` but I am not able to mount it.
3. Is it the shared file system problem caused `s3` repository registry failed in K8S/OCP environment?