# Cannot register a fs snapshot repository in elastic 8.13.0
**URL:** https://discuss.elastic.co/t/cannot-register-a-fs-snapshot-repository-in-elastic-8-13-0/357774
**Category:** Elasticsearch
**Tags:** snapshot-and-restore
**Created:** [April 19, 2024, 8:42am UTC](https://discuss.elastic.co/t/cannot-register-a-fs-snapshot-repository-in-elastic-8-13-0/357774 "2024-04-19T08:42:39Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@dove-young](https://discuss.elastic.co/u/dove-young)
#### Post date: [April 19, 2024, 8:42am UTC](https://discuss.elastic.co/t/cannot-register-a-fs-snapshot-repository-in-elastic-8-13-0/357774/1 "2024-04-19T08:42:39Z")
I installed elastic 8.13.0 in OCP 4.15.
I defined `path.repo` to ["/usr/share/elasticsearch/data/backup"]`in`elasticsearch.yml`, but when I try to register a FS repository, I got error like`"master\_not\_discovered\_exception"`
```auto
sh-5.0$ export ELASTIC_PASSWORD="9TcoQN8x5cT641a1cvXR852U"
sh-5.0$ curl -X PUT "http://instana-es-http:9200/_snapshot/my_fs_backup?pretty" -H 'Content-Type: application/json' -u elastic:$ELASTIC_PASSWORD -d'
> {
> "type": "fs",
> "settings": {
> "location": "/usr/share/elasticsearch/data/backup"
> }
> }'
{
"error" : {
"root_cause" : [
{
"type" : "master_not_discovered_exception",
"reason" : null
}
],
"type" : "master_not_discovered_exception",
"reason" : null
},
"status" : 503
}
```
---
### Author:  [@dove-young](https://discuss.elastic.co/u/dove-young)
#### Post date: [April 19, 2024, 8:45am UTC](https://discuss.elastic.co/t/cannot-register-a-fs-snapshot-repository-in-elastic-8-13-0/357774/2 "2024-04-19T08:45:03Z")
I created the elasticsearch CR in this way. I cannot un-comment `node.master: true` and `node.data: true`. Otherwise elastic pod will keep crash.
```auto
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: instana
spec:
version: 8.13.0
nodeSets:
- name: default
count: 3
config:
# node.master: true
# node.data: true
# node.ingest: true
node.store.allow_mmap: false
podTemplate:
spec:
imagePullSecrets:
- name: docker-registry
volumeClaimTemplates:
- metadata:
name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
http:
tls:
selfSignedCertificate:
disabled: true
```
---
### Author:  [@dove-young](https://discuss.elastic.co/u/dove-young)
#### Post date: [April 19, 2024, 8:46am UTC](https://discuss.elastic.co/t/cannot-register-a-fs-snapshot-repository-in-elastic-8-13-0/357774/3 "2024-04-19T08:46:25Z")
Also I am not able to attach an NFS volume to elastic STS. The pod will not be created and describe sts said it was forbidden by security constraint in elastic installation.
```auto
Warning: would violate PodSecurity "restricted:v1.24": restricted volume types (volume "nfs-vol" uses restricted volume type "nfs"), seccompProfile (pod or containers "elastic-internal-init-filesystem", "elastic-internal-suspend", "elasticsearch" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
Warning FailedCreate 109s (x3 over 109s) statefulset-controller create Pod instana-es-default-0 in StatefulSet instana-es-default failed error: pods "instana-es-default-0" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, spec.volumes[1]: Invalid value: "nfs": nfs volumes are not allowed to be used, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "cassandra-scc": Forbidden: not usable by user or serviceaccount, provider "clickhouse-scc": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "rook-ceph": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "rook-ceph-csi": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]
```
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [April 19, 2024, 8:54am UTC](https://discuss.elastic.co/t/cannot-register-a-fs-snapshot-repository-in-elastic-8-13-0/357774/4 "2024-04-19T08:54:31Z")
> [@dove-young](#):
>
> `master_not_discovered_exception`
Your cluster has more fundamental problems than the ones related to snapshots. See [these docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/discovery-troubleshooting.html) for more information:
> If your cluster doesn’t have a stable master, many of its features won’t work correctly and Elasticsearch will report errors to clients and in its logs. You must fix the master node’s instability before addressing these other issues. It will not be possible to solve any other issues while there is no elected master node or the elected master node is unstable.
---
### Author:  [@dove-young](https://discuss.elastic.co/u/dove-young)
#### Post date: [April 19, 2024, 9:55am UTC](https://discuss.elastic.co/t/cannot-register-a-fs-snapshot-repository-in-elastic-8-13-0/357774/5 "2024-04-19T09:55:25Z")
I un-comment the `node.master: true` and `node.data: true` in my CR, then I got these logs in my pod logs.
it says that `unknown setting [node.data] please check that any required plugins are installed`. Is it true there is NO plugins in elastic v8.13.x?
And is there any document tell me what exactly items should be in a elastic CR and what exactly items should be in `elasticsearch.yaml` ?
```auto
{"@timestamp":"2024-04-19T09:51:11.378Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"instana-es-default-0","elasticsearch.cluster.name":"instana","error.type":"java.lang.IllegalArgumentException","error.message":"unknown setting [node.data] please check that any required plugins are installed, or check the breaking changes documentation for removed settings","error.stack_trace":"java.lang.IllegalArgumentException: unknown setting [node.data] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:563)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:509)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:479)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.common.settings.SettingsModule.(SettingsModule.java:132)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.common.settings.SettingsModule.(SettingsModule.java:50)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.node.NodeConstruction.validateSettings(NodeConstruction.java:494)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:255)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.node.Node.(Node.java:192)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.bootstrap.Elasticsearch$2.(Elasticsearch.java:237)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:237)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:74)\n\tSuppressed: java.lang.IllegalArgumentException: unknown setting [node.master] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\t\t... 12 more\n"}
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/instana.log
ERROR: Elasticsearch died while starting up, with exit code 1
```
---
### Author:  [@dove-young](https://discuss.elastic.co/u/dove-young)
#### Post date: [April 19, 2024, 9:59am UTC](https://discuss.elastic.co/t/cannot-register-a-fs-snapshot-repository-in-elastic-8-13-0/357774/6 "2024-04-19T09:59:09Z")
If I un-comment `node.master: true` only, the pod log will said `unknown setting [node.master]`
```auto
{"@timestamp":"2024-04-19T09:57:21.805Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"instana-es-default-0","elasticsearch.cluster.name":"instana","error.type":"java.lang.IllegalArgumentException","error.message":"unknown setting [node.master] please check that any required plugins are installed, or check the breaking changes documentation for removed settings","error.stack_trace":"java.lang.IllegalArgumentException: unknown setting [node.master] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:563)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:509)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:479)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.common.settings.SettingsModule.(SettingsModule.java:132)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.common.settings.SettingsModule.(SettingsModule.java:50)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.node.NodeConstruction.validateSettings(NodeConstruction.java:494)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:255)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.node.Node.(Node.java:192)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.bootstrap.Elasticsearch$2.(Elasticsearch.java:237)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:237)\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:74)\n"}
```
---
### Author:  [@dove-young](https://discuss.elastic.co/u/dove-young)
#### Post date: [April 19, 2024, 10:33am UTC](https://discuss.elastic.co/t/cannot-register-a-fs-snapshot-repository-in-elastic-8-13-0/357774/7 "2024-04-19T10:33:57Z")
I got clue. When I installed elasticsearch, it works fine and elected master.
then when I changed `elasticsearch.yml` to. add `path.repo` by change the secret by which elasticsearch.yml mounted, then restarted all the 3 elastic pod, all pods keep claim no master been discovered.
```auto
{"@timestamp":"2024-04-19T10:23:28.349Z", "log.level": "WARN", "message":"master not discovered or elected yet, an election requires at least 2 nodes with ids from [NXbb5SLjRW6zTsk5w3_Hqg, cgMQ3BlcSASi0Abi-do6cg, 9F7Mj-brSxmqMINmgtf8Sg], have only discovered non-quorum [{instana-es-default-0}{9F7Mj-brSxmqMINmgtf8Sg}{shoCtvUDTKef6e1Uzategw}{instana-es-default-0}{10.254.32.59}{10.254.32.59:9300}{cdfhilmrstw}{8.13.0}{7000099-8503000}]; discovery will continue using [10.254.12.53:9300, 10.254.28.54:9300, 10.254.32.58:9300] from hosts providers and [{instana-es-default-0}{9F7Mj-brSxmqMINmgtf8Sg}{shoCtvUDTKef6e1Uzategw}{instana-es-default-0}{10.254.32.59}{10.254.32.59:9300}{cdfhilmrstw}{8.13.0}{7000099-8503000}] from last-known cluster state; node term 1, last-accepted version 55 in term 1; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.13/discovery-troubleshooting.html", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[instana-es-default-0][cluster_coordination][T#1]","log.logger":"org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper","elasticsearch.node.name":"instana-es-default-0","elasticsearch.cluster.name":"instana"}
```
---
### Author:  [@dove-young](https://discuss.elastic.co/u/dove-young)
#### Post date: [April 19, 2024, 10:41am UTC](https://discuss.elastic.co/t/cannot-register-a-fs-snapshot-repository-in-elastic-8-13-0/357774/8 "2024-04-19T10:41:23Z")
waiting for a while, I can find these messages from pod log
```auto
{"@timestamp":"2024-04-19T10:34:18.937Z", "log.level": "WARN", "message":"address [10.254.12.53:9300], node [unknown discovery result: [][10.254.12.53:9300] connect_timeout[30s]; for summary, see logs from org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.13/discovery-troubleshooting.html", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[instana-es-default-0][generic][T#6]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.node.name":"instana-es-default-0","elasticsearch.cluster.name":"instana"}
{"@timestamp":"2024-04-19T10:34:18.937Z", "log.level": "WARN", "message":"address [10.254.28.54:9300], node [unknown discovery result: [][10.254.28.54:9300] connect_timeout[30s]; for summary, see logs from org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.13/discovery-troubleshooting.html", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[instana-es-default-0][generic][T#5]","log.logger":"org.elasticsearch.discovery.PeerFinde
r","elasticsearch.node.name":"instana-es-default-0","elasticsearch.cluster.name":"instana"}
{"@timestamp":"2024-04-19T10:34:18.937Z", "log.level": "WARN", "message":"address [10.254.32.58:9300], node [unknown discovery result: [][10.254.32.58:9300] connect_timeout[30s]; for summary, see logs from org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.13/discovery-troubleshooting.html", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[instana-es-default-0][generic][T#3]","log.logger":"org.elasticsearch.discovery.PeerFinde
r","elasticsearch.node.name":"instana-es-default-0","elasticsearch.cluster.name":"instana"}
```
but there was no those IPs actually after restart
```auto
oc get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
curl 0/1 Completed 0 2d1h 10.254.16.27 worker2.o1-729588
instana-es-default-0 1/1 Running 0 10m 10.254.32.60 worker3.o1-729588
instana-es-default-1 1/1 Running 0 10m 10.254.28.56 worker4.o1-729588
instana-es-default-2 1/1 Running 0 10m 10.254.12.55 worker0.o1-729588
```
---
### Author:  [@dove-young](https://discuss.elastic.co/u/dove-young)
#### Post date: [April 19, 2024, 11:00am UTC](https://discuss.elastic.co/t/cannot-register-a-fs-snapshot-repository-in-elastic-8-13-0/357774/9 "2024-04-19T11:00:58Z")
I tried another approach. I scale the elastic sts to 0 replicas, then scale it to 3, but elastic pod still looking to the old IP addresses for nodes. That should be a problem.
```auto
[root@api.o1-729588 ~]# oc scale sts instana-es-default --replicas=0
statefulset.apps/instana-es-default scaled
[root@api.o1-729588 ~]# oc scale sts instana-es-default --replicas=3
statefulset.apps/instana-es-default scaled
[root@api.o1-729588 ~]#
[root@api.o1-729588 ~]# oc get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
curl 0/1 Completed 0 2d1h 10.254.16.27 worker2.o1-729588
instana-es-default-0 1/1 Running 0 108s 10.254.32.62 worker3.o1-729588
instana-es-default-1 1/1 Running 0 108s 10.254.12.58 worker0.o1-729588
instana-es-default-2 1/1 Running 0 108s 10.254.28.58 worker4.o1-729588
[root@api.o1-729588 ~]#
```
```auto
{"@timestamp":"2024-04-19T11:04:41.788Z", "log.level": "WARN", "message":"address [10.254.12.57:9300], node [unknown discovery result: [][10.254.12.57:9300] connect_timeout[30s]; for summary, see logs from org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.13/discovery-troubleshooting.html", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[instana-es-default-0][generic][T#3]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.node.name":"instana-es-default-0","elasticsearch.cluster.name":"instana"}
{"@timestamp":"2024-04-19T11:04:41.788Z", "log.level": "WARN", "message":"address [10.254.32.61:9300], node [unknown discovery result: [][10.254.32.61:9300] connect_timeout[30s]; for summary, see logs from org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.13/discovery-troubleshooting.html", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[instana-es-default-0][generic][T#2]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.node.name":"instana-es-default-0","elasticsearch.cluster.name":"instana"}
{"@timestamp":"2024-04-19T11:04:41.788Z", "log.level": "WARN", "message":"address [10.254.28.57:9300], node [unknown discovery result: [][10.254.28.57:9300] connect_timeout[30s]; for summary, see logs from org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.13/discovery-troubleshooting.html", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[instana-es-default-0][generic][T#5]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.node.name":"instana-es-default-0","elasticsearch.cluster.name":"instana"}
{"@timestamp":"2024-04-19T11:04:50.706Z", "log.level": "WARN", "message":"master not discovered or elected yet, an election requires at least 2 nodes with ids from [DXo4BIiJTRSUOiGCCDy9Zw, 8zZQRMjhQRq8zdzW7CEgoQ, l8imZI5dSOKbHNCIIo_i_A], have only discovered non-quorum [{instana-es-default-0}{l8imZI5dSOKbHNCIIo_i_A}{2pluVwLARqGKTcfo7P1Gvg}{instana-es-default-0}{10.254.32.62}{10.254.32.62:9300}{cdfhilmrstw}{8.13.0}{7000099-8503000}]; discovery will continue using [10.254.12.57:9300, 10.254.28.57:9300, 10.254.32.61:9300] from hosts providers and [{instana-es-default-0}{l8imZI5dSOKbHNCIIo_i_A}{2pluVwLARqGKTcfo7P1Gvg}{instana-es-default-0}{10.254.32.62}{10.254.32.62:9300}{cdfhilmrstw}{8.13.0}{7000099-8503000}] from last-known cluster state; node term 3, last-accepted version 58 in term 3; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.13/discovery-troubleshooting.html", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[instana-es-default-0][cluster_coordination][T#1]","log.logger":"org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper","elasticsearch.node.name":"instana-es-default-0","elasticsearch.cluster.name":"instana"}
```
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [April 19, 2024, 11:52am UTC](https://discuss.elastic.co/t/cannot-register-a-fs-snapshot-repository-in-elastic-8-13-0/357774/10 "2024-04-19T11:52:19Z")
> [@dove-young](#):
>
> it says that `unknown setting [node.data] please check that any required plugins are installed`. Is it true there is NO plugins in elastic v8.13.x?
That's right, `node.data` and `node.master` aren't settings any more. See [these docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#node-roles) for more information.