Creating an Elasticsearch resource in Minikube multi-node cluster fails

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

What did you do?
Created an Elasticsearch resource on a multi-node Minikube cluster:

$ minikube start -n 4 --cni=calico
$ apply -f https://download.elastic.co/downloads/eck/2.9.0/crds.yaml
$ apply -f https://download.elastic.co/downloads/eck/2.9.0/operator.yaml
$ apply -f elasticsearch-with-service.yaml (file content in "Resource definition" section below)

What did you expect to see?
The created quickstart-es-default-0 pod logs won't contain errors.

What did you see instead? Under which circumstances?
The created quickstart-es-default-0 pod logs contain errors.

Environment

  • ECK version: 2.9.0

  • Kubernetes information:

    • Kubernetes distribution: Minikube
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:20:54Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-14T09:47:40Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}

$ minikube version --components
minikube version: v1.31.1
commit: fd3f3801765d093a485d255043149f92ec0a695f

buildctl:
buildctl github.com/moby/buildkit v0.11.6 2951a28cd7085eb18979b1f710678623d94ed578

containerd:
containerd containerd.io 1.6.21 3dce8eb055cbb6872793272b4f20ed16117344f8

crictl:
crictl version v1.25.0

crio:
crio version 1.24.6

crun:
crun version UNKNOWN

ctr:
ctr containerd.io 1.6.21

docker:
Docker version 24.0.4, build 3713ee1

dockerd:
Docker version 24.0.4, build 4ffc614

podman:
podman version 3.4.4

runc:
runc version 1.1.7

$ uname -s -r -v -m
Linux 5.15.0-50-generic #56~20.04.1-Ubuntu SMP Tue Sep 27 15:51:29 UTC 2022 x86_64

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.6 LTS
Release:	20.04
Codename:	focal
  • Resource definition:
https://download.elastic.co/downloads/eck/2.9.0/crds.yaml
https://download.elastic.co/downloads/eck/2.9.0/operator.yaml

elasticsearch-with-service.yaml:
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 8.9.0
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false
  http:
    service:
      spec:
        type: LoadBalancer
  • Logs:
$ kubectl logs quickstart-es-default-0
Defaulted container "elasticsearch" out of: elasticsearch, elastic-internal-init-filesystem (init), elastic-internal-suspend (init)
Skipping security auto configuration because the configuration file [/usr/share/elasticsearch/config/elasticsearch.yml] is missing or is not a regular file
{"@timestamp":"2023-08-04T17:28:44.601Z", "log.level": "INFO", "message":"Java vector incubator API enabled; uses preferredBitSize=256", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.apache.lucene.util.VectorUtilPanamaProvider","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
...
{"@timestamp":"2023-08-04T17:28:55.484Z", "log.level": "INFO", "message":"loaded module [x-pack-eql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"timestamp": "2023-08-04T17:28:59+00:00", "message": "readiness probe failed", "curl_rc": "7"}
{"timestamp": "2023-08-04T17:29:04+00:00", "message": "readiness probe failed", "curl_rc": "7"}
{"@timestamp":"2023-08-04T17:29:08.750Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart","error.type":"java.lang.IllegalStateException","error.message":"failed to obtain node locks, tried [/usr/share/elasticsearch/data]; maybe these locations are not writable or multiple nodes were started on the same data path?","error.stack_trace":"java.lang.IllegalStateException: failed to obtain node locks, tried [/usr/share/elasticsearch/data]; maybe these locations are not writable or multiple nodes were started on the same data path?\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:292)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.node.Node.<init>(Node.java:490)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.node.Node.<init>(Node.java:334)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:234)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:234)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:72)\nCaused by: java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:237)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:205)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:284)\n\t... 5 more\nCaused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/data/node.lock\n\tat java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)\n\tat java.base/sun.nio.fs.UnixPath.toRealPath(UnixPath.java:833)\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:94)\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:43)\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:44)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:230)\n\t... 7 more\n\tSuppressed: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/node.lock\n\t\tat java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)\n\t\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)\n\t\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)\n\t\tat java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:261)\n\t\tat java.base/java.nio.file.Files.newByteChannel(Files.java:379)\n\t\tat java.base/java.nio.file.Files.createFile(Files.java:657)\n\t\tat org.apache.lucene.core@9.7.0/org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:84)\n\t\t... 10 more\n"}
# Readable error stack-trace added on the code block below
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/quickstart.log
{"timestamp": "2023-08-04T17:29:08+00:00", "message": "readiness probe failed", "curl_rc": "7"}

ERROR: Elasticsearch exited unexpectedly

Readable error stack-trace:

java.lang.IllegalStateException: failed to obtain node locks, tried [/usr/share/elasticsearch/data]; maybe these locations are not writable or multiple nodes were started on the same data path?
	at org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:292)
	at org.elasticsearch.server@8.9.0/org.elasticsearch.node.Node.<init>(Node.java:490)
	at org.elasticsearch.server@8.9.0/org.elasticsearch.node.Node.<init>(Node.java:334)
	at org.elasticsearch.server@8.9.0/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:234)
	at org.elasticsearch.server@8.9.0/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:234)
	at org.elasticsearch.server@8.9.0/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:72)
Caused by: java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data
	at org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:237)
	at org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:205)
	at org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:284)
	... 5 more
Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/data/node.lock
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixPath.toRealPath(UnixPath.java:833)
	at org.apache.lucene.core@9.7.0/org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:94)
	at org.apache.lucene.core@9.7.0/org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:43)
	at org.apache.lucene.core@9.7.0/org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:44)
	at org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:230)
	... 7 more
	Suppressed: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/node.lock
		at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
		at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
		at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
		at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:261)
		at java.base/java.nio.file.Files.newByteChannel(Files.java:379)
		at java.base/java.nio.file.Files.createFile(Files.java:657)
		at org.apache.lucene.core@9.7.0/org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:84)
		... 10 more
3

As suggested in the error message,

there's a known bug in Minikube for Persistent Volumes permission issue for 2+ node clutsters.

Following this comment fixed me this issue.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.