Failed to obtain node locks, tried [/usr/share/elasticsearch/data]

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hello Team,
I have installed ECK Operator(v2.9.0) on openshift(v 4.13.18). Installation of the operator is successful. If I am creating elasticsearch cluster via an operator, pods are getting crashed with following error:

{"@timestamp":"2023-11-06T19:10:13.449Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart","error.type":"java.lang.IllegalStateException","error.message":"failed to obtain node locks, tried [/usr/share/elasticsearch/data]; maybe these locations are not writable or multiple nodes were started on the same data path?","error.stack_trace":"java.lang.IllegalStateException: failed to obtain node locks, tried [/usr/share/elasticsearch/data]; maybe these locations are not writable or multiple nodes were started on the same data path?\n\tat org.elasticsearch.server@8.10.4/org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:292)\n\tat org.elasticsearch.server@8.10.4/org.elasticsearch.node.Node.(Node.java:490)\n\tat org.elasticsearch.server@8.10.4/org.elasticsearch.node.Node.(Node.java:338)\n\tat org.elasticsearch.server@8.10.4/org.elasticsearch.bootstrap.Elasticsearch$2.(Elasticsearch.java:234)\n\tat org.elasticsearch.server@8.10.4/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:234)\n\tat org.elasticsearch.server@8.10.4/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:72)\nCaused by: java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data\n\tat org.elasticsearch.server@8.10.4/org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:237)\n\tat org.elasticsearch.server@8.10.4/org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:205)\n\tat org.elasticsearch.server@8.10.4/org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:284)\n\t... 5 more\nCaused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/data/node.lock\n\tat java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)\n\tat java.base/sun.nio.fs.UnixPath.toRealPath(UnixPath.java:834)\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:94)\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:43)\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:44)\n\tat org.elasticsearch.server@8.10.4/org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:230)\n\t... 7 more\n\tSuppressed: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/node.lock\n\t\tat java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)\n\t\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)\n\t\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)\n\t\tat java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:261)\n\t\tat java.base/java.nio.file.Files.newByteChannel(Files.java:379)\n\t\tat java.base/java.nio.file.Files.createFile(Files.java:657)\n\t\tat org.apache.lucene.core@9.7.0/org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:84)\n\t\t... 10 more\n"}
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/quickstart.log

ERROR: Elasticsearch exited unexpectedly, with exit code 1

What did you expect to see?
Should be able to install Elasticsearch cluster
What did you see instead? Under which circumstances?
We have two cluster. We see this issue in only one cluster.
Environment

  • ECK version:
    2.9.0
  • Kubernetes information
    • Kubernetes distribution: Openshift on AWS(4.13.18)
  • Resource definition:
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 8.10.4
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false

Can you please help here? Thanks

2

Hi,

Did you check these suggestions?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.