What did you do?
Created an Elasticsearch resource on a multi-node Minikube cluster:
$ minikube start -n 4 --cni=calico
$ apply -f https://download.elastic.co/downloads/eck/2.9.0/crds.yaml
$ apply -f https://download.elastic.co/downloads/eck/2.9.0/operator.yaml
$ apply -f elasticsearch-with-service.yaml (file content in "Resource definition" section below)
What did you expect to see?
The created quickstart-es-default-0 pod logs won't contain errors.
What did you see instead? Under which circumstances?
The created quickstart-es-default-0 pod logs contain errors.
Environment
-
ECK version: 2.9.0
-
Kubernetes information:
- Kubernetes distribution: Minikube
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:20:54Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-14T09:47:40Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}
$ minikube version --components
minikube version: v1.31.1
commit: fd3f3801765d093a485d255043149f92ec0a695f
buildctl:
buildctl github.com/moby/buildkit v0.11.6 2951a28cd7085eb18979b1f710678623d94ed578
containerd:
containerd containerd.io 1.6.21 3dce8eb055cbb6872793272b4f20ed16117344f8
crictl:
crictl version v1.25.0
crio:
crio version 1.24.6
crun:
crun version UNKNOWN
ctr:
ctr containerd.io 1.6.21
docker:
Docker version 24.0.4, build 3713ee1
dockerd:
Docker version 24.0.4, build 4ffc614
podman:
podman version 3.4.4
runc:
runc version 1.1.7
$ uname -s -r -v -m
Linux 5.15.0-50-generic #56~20.04.1-Ubuntu SMP Tue Sep 27 15:51:29 UTC 2022 x86_64
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.6 LTS
Release: 20.04
Codename: focal
- Resource definition:
https://download.elastic.co/downloads/eck/2.9.0/crds.yaml
https://download.elastic.co/downloads/eck/2.9.0/operator.yaml
elasticsearch-with-service.yaml:
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: quickstart
spec:
version: 8.9.0
nodeSets:
- name: default
count: 1
config:
node.store.allow_mmap: false
http:
service:
spec:
type: LoadBalancer
- Logs:
$ kubectl logs quickstart-es-default-0
Defaulted container "elasticsearch" out of: elasticsearch, elastic-internal-init-filesystem (init), elastic-internal-suspend (init)
Skipping security auto configuration because the configuration file [/usr/share/elasticsearch/config/elasticsearch.yml] is missing or is not a regular file
{"@timestamp":"2023-08-04T17:28:44.601Z", "log.level": "INFO", "message":"Java vector incubator API enabled; uses preferredBitSize=256", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.apache.lucene.util.VectorUtilPanamaProvider","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
...
{"@timestamp":"2023-08-04T17:28:55.484Z", "log.level": "INFO", "message":"loaded module [x-pack-eql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}
{"timestamp": "2023-08-04T17:28:59+00:00", "message": "readiness probe failed", "curl_rc": "7"}
{"timestamp": "2023-08-04T17:29:04+00:00", "message": "readiness probe failed", "curl_rc": "7"}
{"@timestamp":"2023-08-04T17:29:08.750Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart","error.type":"java.lang.IllegalStateException","error.message":"failed to obtain node locks, tried [/usr/share/elasticsearch/data]; maybe these locations are not writable or multiple nodes were started on the same data path?","error.stack_trace":"java.lang.IllegalStateException: failed to obtain node locks, tried [/usr/share/elasticsearch/data]; maybe these locations are not writable or multiple nodes were started on the same data path?\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:292)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.node.Node.<init>(Node.java:490)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.node.Node.<init>(Node.java:334)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:234)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:234)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:72)\nCaused by: java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:237)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:205)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:284)\n\t... 5 more\nCaused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/data/node.lock\n\tat java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)\n\tat java.base/sun.nio.fs.UnixPath.toRealPath(UnixPath.java:833)\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:94)\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:43)\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:44)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:230)\n\t... 7 more\n\tSuppressed: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/node.lock\n\t\tat java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)\n\t\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)\n\t\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)\n\t\tat java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:261)\n\t\tat java.base/java.nio.file.Files.newByteChannel(Files.java:379)\n\t\tat java.base/java.nio.file.Files.createFile(Files.java:657)\n\t\tat org.apache.lucene.core@9.7.0/org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:84)\n\t\t... 10 more\n"}
# Readable error stack-trace added on the code block below
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/quickstart.log
{"timestamp": "2023-08-04T17:29:08+00:00", "message": "readiness probe failed", "curl_rc": "7"}
ERROR: Elasticsearch exited unexpectedly
Readable error stack-trace:
java.lang.IllegalStateException: failed to obtain node locks, tried [/usr/share/elasticsearch/data]; maybe these locations are not writable or multiple nodes were started on the same data path?
at org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:292)
at org.elasticsearch.server@8.9.0/org.elasticsearch.node.Node.<init>(Node.java:490)
at org.elasticsearch.server@8.9.0/org.elasticsearch.node.Node.<init>(Node.java:334)
at org.elasticsearch.server@8.9.0/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:234)
at org.elasticsearch.server@8.9.0/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:234)
at org.elasticsearch.server@8.9.0/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:72)
Caused by: java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data
at org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:237)
at org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:205)
at org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:284)
... 5 more
Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/data/node.lock
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixPath.toRealPath(UnixPath.java:833)
at org.apache.lucene.core@9.7.0/org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:94)
at org.apache.lucene.core@9.7.0/org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:43)
at org.apache.lucene.core@9.7.0/org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:44)
at org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:230)
... 7 more
Suppressed: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/node.lock
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:261)
at java.base/java.nio.file.Files.newByteChannel(Files.java:379)
at java.base/java.nio.file.Files.createFile(Files.java:657)
at org.apache.lucene.core@9.7.0/org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:84)
... 10 more