# Creating an Elasticsearch resource in Minikube multi-node cluster fails **URL:** \ **Category:** Elastic Cloud on Kubernetes (ECK)\ **Tags:** docker\ **Created:** [August 5, 2023, 9:26pm UTC](https://discuss.elastic.co/t/creating-an-elasticsearch-resource-in-minikube-multi-node-cluster-fails/340184 "2023-08-05T21:26:05Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![wh1te\_rabb1t](https://avatars.discourse-cdn.com/v4/letter/w/9d8465/32.png) [@wh1te\_rabb1t](https://discuss.elastic.co/u/wh1te_rabb1t)\ **Post date:** [August 5, 2023, 9:26pm UTC](https://discuss.elastic.co/t/creating-an-elasticsearch-resource-in-minikube-multi-node-cluster-fails/340184/1 "2023-08-05T21:26:05Z")
**What did you do?** Created an `Elasticsearch` resource on a multi-node Minikube cluster: ```auto $ minikube start -n 4 --cni=calico $ apply -f https://download.elastic.co/downloads/eck/2.9.0/crds.yaml $ apply -f https://download.elastic.co/downloads/eck/2.9.0/operator.yaml $ apply -f elasticsearch-with-service.yaml (file content in "Resource definition" section below) ``` **What did you expect to see?** The created `quickstart-es-default-0` pod logs won't contain errors. **What did you see instead? Under which circumstances?** The created `quickstart-es-default-0` pod logs contain errors. **Environment** - ECK version: 2.9.0 - Kubernetes information: ```auto $ kubectl version Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:20:54Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v5.0.1 Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-14T09:47:40Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"} ``` ```auto $ minikube version --components minikube version: v1.31.1 commit: fd3f3801765d093a485d255043149f92ec0a695f buildctl: buildctl github.com/moby/buildkit v0.11.6 2951a28cd7085eb18979b1f710678623d94ed578 containerd: containerd containerd.io 1.6.21 3dce8eb055cbb6872793272b4f20ed16117344f8 crictl: crictl version v1.25.0 crio: crio version 1.24.6 crun: crun version UNKNOWN ctr: ctr containerd.io 1.6.21 docker: Docker version 24.0.4, build 3713ee1 dockerd: Docker version 24.0.4, build 4ffc614 podman: podman version 3.4.4 runc: runc version 1.1.7 ``` ```auto $ uname -s -r -v -m Linux 5.15.0-50-generic #56~20.04.1-Ubuntu SMP Tue Sep 27 15:51:29 UTC 2022 x86_64 ``` ```auto $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.6 LTS Release: 20.04 Codename: focal ``` - Resource definition: ```auto https://download.elastic.co/downloads/eck/2.9.0/crds.yaml https://download.elastic.co/downloads/eck/2.9.0/operator.yaml elasticsearch-with-service.yaml: apiVersion: elasticsearch.k8s.elastic.co/v1 kind: Elasticsearch metadata: name: quickstart spec: version: 8.9.0 nodeSets: - name: default count: 1 config: node.store.allow_mmap: false http: service: spec: type: LoadBalancer ``` - Logs: ```auto $ kubectl logs quickstart-es-default-0 Defaulted container "elasticsearch" out of: elasticsearch, elastic-internal-init-filesystem (init), elastic-internal-suspend (init) Skipping security auto configuration because the configuration file [/usr/share/elasticsearch/config/elasticsearch.yml] is missing or is not a regular file {"@timestamp":"2023-08-04T17:28:44.601Z", "log.level": "INFO", "message":"Java vector incubator API enabled; uses preferredBitSize=256", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.apache.lucene.util.VectorUtilPanamaProvider","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"} ... {"@timestamp":"2023-08-04T17:28:55.484Z", "log.level": "INFO", "message":"loaded module [x-pack-eql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"} {"timestamp": "2023-08-04T17:28:59+00:00", "message": "readiness probe failed", "curl_rc": "7"} {"timestamp": "2023-08-04T17:29:04+00:00", "message": "readiness probe failed", "curl_rc": "7"} {"@timestamp":"2023-08-04T17:29:08.750Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart","error.type":"java.lang.IllegalStateException","error.message":"failed to obtain node locks, tried [/usr/share/elasticsearch/data]; maybe these locations are not writable or multiple nodes were started on the same data path?","error.stack_trace":"java.lang.IllegalStateException: failed to obtain node locks, tried [/usr/share/elasticsearch/data]; maybe these locations are not writable or multiple nodes were started on the same data path?\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:292)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.node.Node.(Node.java:490)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.node.Node.(Node.java:334)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.bootstrap.Elasticsearch$2.(Elasticsearch.java:234)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:234)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:72)\nCaused by: java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:237)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:205)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:284)\n\t... 5 more\nCaused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/data/node.lock\n\tat java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)\n\tat java.base/sun.nio.fs.UnixPath.toRealPath(UnixPath.java:833)\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:94)\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:43)\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:44)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:230)\n\t... 7 more\n\tSuppressed: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/node.lock\n\t\tat java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)\n\t\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)\n\t\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)\n\t\tat java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:261)\n\t\tat java.base/java.nio.file.Files.newByteChannel(Files.java:379)\n\t\tat java.base/java.nio.file.Files.createFile(Files.java:657)\n\t\tat org.apache.lucene.core@9.7.0/org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:84)\n\t\t... 10 more\n"} # Readable error stack-trace added on the code block below ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/quickstart.log {"timestamp": "2023-08-04T17:29:08+00:00", "message": "readiness probe failed", "curl_rc": "7"} ERROR: Elasticsearch exited unexpectedly ``` Readable error stack-trace: ```auto java.lang.IllegalStateException: failed to obtain node locks, tried [/usr/share/elasticsearch/data]; maybe these locations are not writable or multiple nodes were started on the same data path? at org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:292) at org.elasticsearch.server@8.9.0/org.elasticsearch.node.Node.(Node.java:490) at org.elasticsearch.server@8.9.0/org.elasticsearch.node.Node.(Node.java:334) at org.elasticsearch.server@8.9.0/org.elasticsearch.bootstrap.Elasticsearch$2.(Elasticsearch.java:234) at org.elasticsearch.server@8.9.0/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:234) at org.elasticsearch.server@8.9.0/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:72) Caused by: java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data at org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:237) at org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:205) at org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:284) ... 5 more Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/data/node.lock at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixPath.toRealPath(UnixPath.java:833) at org.apache.lucene.core@9.7.0/org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:94) at org.apache.lucene.core@9.7.0/org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:43) at org.apache.lucene.core@9.7.0/org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:44) at org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:230) ... 7 more Suppressed: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/node.lock at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:261) at java.base/java.nio.file.Files.newByteChannel(Files.java:379) at java.base/java.nio.file.Files.createFile(Files.java:657) at org.apache.lucene.core@9.7.0/org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:84) ... 10 more ``` ---
**Author:** ![wh1te\_rabb1t](https://avatars.discourse-cdn.com/v4/letter/w/9d8465/32.png) [@wh1te\_rabb1t](https://discuss.elastic.co/u/wh1te_rabb1t)\ **Post date:** [August 29, 2023, 3:52pm UTC](https://discuss.elastic.co/t/creating-an-elasticsearch-resource-in-minikube-multi-node-cluster-fails/340184/3 "2023-08-29T15:52:10Z")
As suggested in the error message, there's a known bug in Minikube for [Persistent Volumes permission issue for 2+ node clutsters](https://github.com/kubernetes/minikube/issues/12360). Following [this comment](https://github.com/kubernetes/minikube/issues/12360#issuecomment-1430243861) fixed me this issue. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 26, 2023, 3:53pm UTC](https://discuss.elastic.co/t/creating-an-elasticsearch-resource-in-minikube-multi-node-cluster-fails/340184/4 "2023-09-26T15:53:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.