Snapshot repository-s3 registry failed in 7.17

Elastic Stack Elasticsearch
,
1

I run elasticsearch docker container as described in quickstart Quick start | Elasticsearch Guide [7.17] | Elastic,

then I installed repository-s3 plugin and restarted container.

I have a local minio server act as S3 storage. But I am not able to registry a S3 repository in Elasticsearch. The error msg said "reason" : "Failed to connect to service endpoint: ", but it does not indicate what service endpoint it connected to.

[root@st2-tunnel-dev1]# docker exec -it es01-test /bin/bash
root@03caafc38d92:/usr/share/elasticsearch# bin/elasticsearch-keystore add s3.client.default.access_key
Enter value for s3.client.default.access_key:
root@03caafc38d92:/usr/share/elasticsearch# bin/elasticsearch-keystore add s3.client.default.secret_key
Enter value for s3.client.default.secret_key:

root@03caafc38d92:/usr/share/elasticsearch# curl -X PUT "http://localhost:9200/_snapshot/my_s3_repo?pretty" -H 'Content-Type: application/json'  -d'
> {
>   "type": "s3",
>   "settings": {
>     "endpoint": "http://<my-minio-ip>:9000",
>     "protocol": "http",
>     "region": "eu-central-1",
>     "bucket": "elastic-o1-726938",
>     "base_path": "/usr/share/elasticsearch/data/nodes/0",
>     "client": "default",
>     "path_style_access": "true"
>   }
> }'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_s3_repo] path [/usr/share/elasticsearch/data/nodes/0] is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[my_s3_repo] path [/usr/share/elasticsearch/data/nodes/0] is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object [/usr/share/elasticsearch/data/nodes/0/tests-e8hrHXQyQiOIgHh0sypi5w/master.dat] using a single upload",
      "caused_by" : {
        "type" : "sdk_client_exception",
        "reason" : "Failed to connect to service endpoint: ",
        "caused_by" : {
          "type" : "socket_timeout_exception",
          "reason" : "Connect timed out"
        }
      }
    }
  },
  "status" : 500
}

I also tried to adjust endpoint format, then I got error like this

root@03caafc38d92:/usr/share/elasticsearch# curl -X PUT "http://localhost:9200/_snapshot/my_s3_repo?pretty" -H 'Content-Type: application/json'  -d'
{
  "type": "s3",
  "settings": {
    "endpoint": "<my-minio-ip>:9000",
    "protocol": "http",
    "region": "eu-central-1",
    "bucket": "elastic-o1-726938",
    "base_path": "/usr/share/elasticsearch/data/nodes/0",
    "client": "default",
    "path_style_access": "true"
  }
}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_exception",
        "reason" : "[my_s3_repo] Could not determine repository generation from root blobs"
      }
    ],
    "type" : "repository_exception",
    "reason" : "[my_s3_repo] Could not determine repository generation from root blobs",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Exception when listing blobs by prefix [index-]",
      "caused_by" : {
        "type" : "sdk_client_exception",
        "reason" : "Failed to connect to service endpoint: ",
        "caused_by" : {
          "type" : "socket_timeout_exception",
          "reason" : "Connect timed out"
        }
      }
    }
  },
  "status" : 500
}

but I can access to my minio server via curl in this container.

root@03caafc38d92:/usr/share/elasticsearch# curl -k -H 'Host: <my-minio-ip>:9000' -H 'Date: Wed, 29 May 2024 04:58:50 +0000' -H 'Content-Type: application/zstd' -H 'Authorization: AWS minioadmin:18Hr3bux0qN+JM6wxWiKdctEc0k=' http://<my-minio-ip>:9000/
<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4</ID><DisplayName>minio</DisplayName></Owner><Buckets><Bucket><Name>elastic-o1-726938</Name><CreationDate>2024-04-18T05:48:23.891Z</CreationDate></Bucket><Bucket><Name>my-bucket-longterm</Name><CreationDate>2024-04-26T03:03:08.535Z</CreationDate></Bucket><Bucket><Name>mybucket</Name><CreationDate>2024-04-11T10:22:01.282Z</CreationDate></Bucket><Bucket><Name>oadp-o1-726938</Name><CreationDate>2024-04-12T08:39:08.429Z</CreationDate></Bucket><Bucket><Name>rawspanbucket</Name><CreationDate>2024-05-08T05:09:58.909Z</CreationDate></Bucket><Bucket><Name>test-bucket</Name><CreationDate>2024-04-23T04:33:19.262Z</CreationDate></Bucket><Bucket><Name>test-store</Name><CreationDate>2024-04-03T01:27:13.698Z</CreationDate></Bucket><Bucket><Name>veleco-hgliu</Name><CreationDate>2024-04-12T01:54:10.483Z</CreationDate></Bucket><Bucket><Name>veleco-o1-726938</Name><CreationDate>2024-04-12T01:50:04.447Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>root@03caafc38d92:/usr/share/elasticsearch#
root@03caafc38d92:/usr/share/elasticsearch#

if I remove base_path from payload, I got similar error path is not accessible on master node and it also not indicate what path is.

root@03caafc38d92:/usr/share/elasticsearch# curl -X PUT "http://localhost:9200/_snapshot/my_s3_repo2?pretty" -H 'Content-Type: application/json'  -d'
> {
>   "type": "s3",
>   "settings": {
>     "endpoint": "9.112.252.135:9000",
>     "protocol": "http",
>     "region": "eu-central-1",
>     "bucket": "elastic-o1-726938",
>     "client": "default",
>     "path_style_access": "true"
>   }
> }'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_s3_repo2] path  is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[my_s3_repo2] path  is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object [tests-GacOt_JST22bgxGuIZuCBQ/master.dat] using a single upload",
      "caused_by" : {
        "type" : "sdk_client_exception",
        "reason" : "Failed to connect to service endpoint: ",
        "caused_by" : {
          "type" : "socket_timeout_exception",
          "reason" : "Connect timed out"
        }
      }
    }
  },
  "status" : 500
}
2

finally I got it. the base_path must in the path.repo setting in the elasticsearch.yml

root@03caafc38d92:/usr/share/elasticsearch# curl -X PUT "http://localhost:9200/_snapshot/my_s3_repo3?pretty" -H 'Content-Type: application/json'  -d'
> {
>   "type": "s3",
>   "settings": {
>     "endpoint": "<my-minio-ip>:9000",
>     "protocol": "http",
>     "region": "eu-central-1",
>     "bucket": "elastic-o1-726938",
>     "client": "default",
>     "base_path": "/tmp/backup",
>     "path_style_access": "true"
>   }
> }'
{
  "acknowledged" : true
}
root@03caafc38d92:/usr/share/elasticsearch# cat config/elasticsearch.yml
cluster.name: "docker-cluster"
network.host: 0.0.0.0
path.repo: ["/tmp/backup"]
root@03caafc38d92:/usr/share/elasticsearch#
3

But I still not able to register a S3 repository on elasticsearch running in openshift/Kubernetes

[root@st2-tunnel-dev1]# oc rsh -n instana-elastic  instana-es-default-0
Defaulted container "elasticsearch" out of: elasticsearch, elastic-internal-init-filesystem (init), elastic-internal-suspend (init)
sh-5.1$ ELASTIC_PASSWORD=66q45Kg64qd03fbCiVm7y9wj
sh-5.1$ bin/elasticsearch-keystore add s3.client.default.access_key
warning: usage of JAVA_HOME is deprecated, use ES_JAVA_HOME
Enter value for s3.client.default.access_key:
sh-5.1$ bin/elasticsearch-keystore add s3.client.default.secret_key
warning: usage of JAVA_HOME is deprecated, use ES_JAVA_HOME
Enter value for s3.client.default.secret_key:
sh-5.1$
sh-5.1$ curl -X PUT "http://instana-es-http:9200/_snapshot/my_s3_repo?pretty" -H 'Content-Type: application/json' -u elastic:$ELASTIC_PASSWORD -d'
{
  "type": "s3",
  "settings": {
    "endpoint": "<my-minio-ip>:9000",
    "protocol": "http",
    "region": "eu-central-1",
    "bucket": "elastic-o1-726938",
    "base_path": "/tmp",
    "client": "default",
    "path_style_access": "true"
  }
}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_s3_repo] path [/tmp] is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[my_s3_repo] path [/tmp] is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object [/tmp/tests-ajpHGZF5SciLw6Y1XwNP4g/master.dat] using a single upload",
      "caused_by" : {
        "type" : "sdk_client_exception",
        "reason" : "sdk_client_exception: Failed to connect to service endpoint: ",
        "caused_by" : {
          "type" : "i_o_exception",
          "reason" : "Connection refused"
        }
      }
    }
  },
  "status" : 500
}

here is what in config/elasticsearch.yml

path:
    data: /usr/share/elasticsearch/data
    logs: /usr/share/elasticsearch/logs
    repo:
        - /tmp

here is what I defined in elasticsearch CR

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: instana
spec:
  version: 7.17.15
  nodeSets:
    - name: default
      count: 3
      config:
        node.master: true
        node.data: true
        node.ingest: true
        node.store.allow_mmap: false
        path.repo: ["/tmp"]
4

I cannot even create a fs repository in elastic running in OCP/K8S

sh-5.1$ curl -X PUT 'http://instana-es-http:9200/_snapshot/my_fs_2?pretty' -H 'Content-Type: application/json' -u elastic:$ELASTIC_PASSWORD -d '
{
  "type": "fs",
  "settings": {
    "location": "/tmp"
  }
}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_fs_2] [[a-7DjAEeQiWQF5TplyJPyw, 'RemoteTransportException[[instana-es-default-2][10.254.32.66:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_fs_2] a file written by master to the store [/tmp] cannot be accessed on the node [{instana-es-default-2}{a-7DjAEeQiWQF5TplyJPyw}{zbN9rtONQJKogBbnbxzKTA}{10.254.32.66}{10.254.32.66:9300}{cdfhilmrstw}{k8s_node_name=worker3.o1-743345.cp.fyre.ibm.com, ml.machine_memory=2147483648, xpack.installed=true, transform.node=true, ml.max_open_jobs=512, ml.max_jvm_size=1073741824}]. This might indicate that the store [/tmp] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node]; nested: NoSuchFileException[/tmp/tests-XEAkE8J_Sr-b54dzpD-g0Q/master.dat];'], [Waf822duS4ufvW7ryX2PRA, 'RemoteTransportException[[instana-es-default-1][10.254.36.72:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_fs_2] a file written by master to the store [/tmp] cannot be accessed on the node [{instana-es-default-1}{Waf822duS4ufvW7ryX2PRA}{MlJ4UVthSgO_bN4yHGmzDQ}{10.254.36.72}{10.254.36.72:9300}{cdfhilmrstw}{k8s_node_name=worker5.o1-743345.cp.fyre.ibm.com, ml.machine_memory=2147483648, xpack.installed=true, transform.node=true, ml.max_open_jobs=512, ml.max_jvm_size=1073741824}]. This might indicate that the store [/tmp] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node]; nested: NoSuchFileException[/tmp/tests-XEAkE8J_Sr-b54dzpD-g0Q/master.dat];']]"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[my_fs_2] [[a-7DjAEeQiWQF5TplyJPyw, 'RemoteTransportException[[instana-es-default-2][10.254.32.66:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_fs_2] a file written by master to the store [/tmp] cannot be accessed on the node [{instana-es-default-2}{a-7DjAEeQiWQF5TplyJPyw}{zbN9rtONQJKogBbnbxzKTA}{10.254.32.66}{10.254.32.66:9300}{cdfhilmrstw}{k8s_node_name=worker3.o1-743345.cp.fyre.ibm.com, ml.machine_memory=2147483648, xpack.installed=true, transform.node=true, ml.max_open_jobs=512, ml.max_jvm_size=1073741824}]. This might indicate that the store [/tmp] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node]; nested: NoSuchFileException[/tmp/tests-XEAkE8J_Sr-b54dzpD-g0Q/master.dat];'], [Waf822duS4ufvW7ryX2PRA, 'RemoteTransportException[[instana-es-default-1][10.254.36.72:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_fs_2] a file written by master to the store [/tmp] cannot be accessed on the node [{instana-es-default-1}{Waf822duS4ufvW7ryX2PRA}{MlJ4UVthSgO_bN4yHGmzDQ}{10.254.36.72}{10.254.36.72:9300}{cdfhilmrstw}{k8s_node_name=worker5.o1-743345.cp.fyre.ibm.com, ml.machine_memory=2147483648, xpack.installed=true, transform.node=true, ml.max_open_jobs=512, ml.max_jvm_size=1073741824}]. This might indicate that the store [/tmp] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node]; nested: NoSuchFileException[/tmp/tests-XEAkE8J_Sr-b54dzpD-g0Q/master.dat];']]"
  },
  "status" : 500
}
sh-5.1$
5

When I scale down to 1 instace, I can define fs type repository in K8S

[root@st2-tunnel-dev1]#     oc rsh -n instana-elastic  instana-es-default-0
Defaulted container "elasticsearch" out of: elasticsearch, elastic-internal-init-filesystem (init), elastic-internal-suspend (init)
sh-5.1$ ELASTIC_PASSWORD=66q45Kg64qd03fbCiVm7y9wj
sh-5.1$ curl -X PUT 'http://instana-es-http:9200/_snapshot/my_fs_2?pretty' -H 'Content-Type: application/json' -u elastic:$ELASTIC_PASSWORD -d '
{
  "type": "fs",
  "settings": {
    "location": "/tmp"
  }
}'
{
  "acknowledged" : true
}
sh-5.1$

But I still not able to define s3 type registry when there is 1 elastic instance

sh-5.1$ curl -X PUT "http://instana-es-http:9200/_snapshot/my_s3_repo3?pretty" -H 'Content-Type: application/json' -u elastic:$ELASTIC_PASSWORD -d'
{
  "type": "s3",
  "settings": {
    "endpoint": "9.112.252.135:9000",
    "protocol": "http",
    "region": "eu-central-1",
    "bucket": "elastic-o1-726938",
    "base_path": "/tmp",
    "client": "default",
    "path_style_access": "true"
  }
}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_s3_repo3] path [/tmp] is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[my_s3_repo3] path [/tmp] is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object [/tmp/tests-7vIME0laTKufgaxdhlI2PQ/master.dat] using a single upload",
      "caused_by" : {
        "type" : "sdk_client_exception",
        "reason" : "Failed to connect to service endpoint: ",
        "caused_by" : {
          "type" : "connect_exception",
          "reason" : "Connection refused"
        }
      }
    }
  },
  "status" : 500
}
sh-5.1$

when I retry, it output another error mssage

sh-5.1$ curl -X PUT "http://instana-es-http:9200/_snapshot/my_s3_repo3?pretty" -H 'Content-Type: application/json' -u elastic:$ELASTIC_PASSWORD -d'
{
  "type": "s3",
  "settings": {
    "endpoint": "9.112.252.135:9000",
    "protocol": "http",
    "region": "eu-central-1",
    "bucket": "elastic-o1-726938",
    "base_path": "/tmp",
    "client": "default",
    "path_style_access": "true"
  }
}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_exception",
        "reason" : "[my_s3_repo3] Could not determine repository generation from root blobs"
      }
    ],
    "type" : "repository_exception",
    "reason" : "[my_s3_repo3] Could not determine repository generation from root blobs",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Exception when listing blobs by prefix [index-]",
      "caused_by" : {
        "type" : "sdk_client_exception",
        "reason" : "Failed to connect to service endpoint: ",
        "caused_by" : {
          "type" : "connect_exception",
          "reason" : "Connection refused"
        }
      }
    }
  },
  "status" : 500
}
sh-5.1$
6

so here are the questions for elasticsearch 7.17.x installed in K8S/OCP via helm operator version=2.9.0 from elastic https://helm.elastic.co:

  1. Is it true that even in s3 repository registry payload, the base_path also need to be a shared file system between all elastic pods?
  2. How can I add an additional volume to elasticsearch pod in Elasticsearch CR? I searched Elasticsearch CRD, I don't find any volumeMount item in the CRD. I can add a new volume via volumeClaimTemplate but I am not able to mount it.
  3. Is it the shared file system problem caused s3 repository registry failed in K8S/OCP environment?