Hello Team,
I have installed ECK Operator(v2.9.0) on openshift(v 4.13.18). Installation of the operator is successful. If I am creating elasticsearch cluster via an operator, pods are getting crashed with following error:
{"@timestamp":"2023-11-06T19:10:13.449Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart","error.type":"java.lang.IllegalStateException","error.message":"failed to obtain node locks, tried [/usr/share/elasticsearch/data]; maybe these locations are not writable or multiple nodes were started on the same data path?","error.stack_trace":"java.lang.IllegalStateException: failed to obtain node locks, tried [/usr/share/elasticsearch/data]; maybe these locations are not writable or multiple nodes were started on the same data path?\n\tat org.elasticsearch.server@8.10.4/org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:292)\n\tat org.elasticsearch.server@8.10.4/org.elasticsearch.node.Node.(Node.java:490)\n\tat org.elasticsearch.server@8.10.4/org.elasticsearch.node.Node.(Node.java:338)\n\tat org.elasticsearch.server@8.10.4/org.elasticsearch.bootstrap.Elasticsearch$2.(Elasticsearch.java:234)\n\tat org.elasticsearch.server@8.10.4/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:234)\n\tat org.elasticsearch.server@8.10.4/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:72)\nCaused by: java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data\n\tat org.elasticsearch.server@8.10.4/org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:237)\n\tat org.elasticsearch.server@8.10.4/org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:205)\n\tat org.elasticsearch.server@8.10.4/org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:284)\n\t... 5 more\nCaused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/data/node.lock\n\tat java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)\n\tat java.base/sun.nio.fs.UnixPath.toRealPath(UnixPath.java:834)\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:94)\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:43)\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:44)\n\tat org.elasticsearch.server@8.10.4/org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:230)\n\t... 7 more\n\tSuppressed: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/node.lock\n\t\tat java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)\n\t\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)\n\t\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)\n\t\tat java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:261)\n\t\tat java.base/java.nio.file.Files.newByteChannel(Files.java:379)\n\t\tat java.base/java.nio.file.Files.createFile(Files.java:657)\n\t\tat org.apache.lucene.core@9.7.0/org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:84)\n\t\t... 10 more\n"}
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/quickstart.log
ERROR: Elasticsearch exited unexpectedly, with exit code 1
What did you expect to see?
Should be able to install Elasticsearch cluster
What did you see instead? Under which circumstances?
We have two cluster. We see this issue in only one cluster.
Environment
- ECK version:
2.9.0 - Kubernetes information
- Kubernetes distribution: Openshift on AWS(4.13.18)
- Resource definition:
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: quickstart
spec:
version: 8.10.4
nodeSets:
- name: default
count: 1
config:
node.store.allow_mmap: false
Can you please help here? Thanks