# Failed to obtain node locks, tried \[\[/usr/share/elasticsearch/data\]\] with lock id \[0\]

**URL:** <https://discuss.elastic.co/t/failed-to-obtain-node-locks-tried-usr-share-elasticsearch-data-with-lock-id-0/205706>\
**Category:** Elasticsearch\
**Tags:** docker\
**Created:** [October 29, 2019, 3:20pm UTC](https://discuss.elastic.co/t/failed-to-obtain-node-locks-tried-usr-share-elasticsearch-data-with-lock-id-0/205706 "2019-10-29T15:20:58Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Benni](https://avatars.discourse-cdn.com/v4/letter/b/c67d28/32.png) [@Benni](https://discuss.elastic.co/u/Benni)\
**Post date:** [October 29, 2019, 3:20pm UTC](https://discuss.elastic.co/t/failed-to-obtain-node-locks-tried-usr-share-elasticsearch-data-with-lock-id-0/205706/1 "2019-10-29T15:20:58Z")

</div>

Hello, I have already found many approaches for this error. Unfortunately none of them helped me to solve my problem. It seems to be an old friend of Elasticsearch.  
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max\_local\_storage\_nodes] (was [1])?

> Blockquote  
> elasticsearch\_1 | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.  
> elasticsearch\_1 | {"type": "server", "timestamp": "2019-10-29T15:57:33,739+01:00", "level": "WARN", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "elasticsearch", "node.name": "BrewMaster", "message": "uncaught exception in thread [main]",  
> elasticsearch\_1 | "stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max\_local\_storage\_nodes] (was [1])?",  
> .  
> .  
> .  
> elasticsearch\_1 | "Caused by: java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data/nodes/0",  
> elasticsearch\_1 | "at org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]",  
> elasticsearch\_1 | "at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:269) ~[elasticsearch-7.4.1.jar:7.4.1]",  
> elasticsearch\_1 | "at org.elasticsearch.node.Node.(Node.java:275) ~[elasticsearch-7.4.1.jar:7.4.1]",  
> elasticsearch\_1 | "at org.elasticsearch.node.Node.(Node.java:255) ~[elasticsearch-7.4.1.jar:7.4.1]",  
> elasticsearch\_1 | "at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]",  
> elasticsearch\_1 | "at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]",  
> elasticsearch\_1 | "at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.1.jar:7.4.1]",  
> elasticsearch\_1 | "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.1.jar:7.4.1]",  
> elasticsearch\_1 | "... 6 more",  
> elasticsearch\_1 | "Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes/0/node.lock",  
> elasticsearch\_1 | "at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]",  
> elasticsearch\_1 | "at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]",  
> elasticsearch\_1 | "at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]",  
> elasticsearch\_1 | "at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:182) ~[?:?]",  
> elasticsearch\_1 | "at java.nio.channels.FileChannel.open(FileChannel.java:292) ~[?:?]",  
> elasticsearch\_1 | "at java.nio.channels.FileChannel.open(FileChannel.java:345) ~[?:?]",  
> .  
> .  
> .

This is the error I get and this is how my docker file, composite file and my .yml file looks like.

```
dockerfile-compose.yml:
elasticsearch:
build:
  context: elasticsearch
ports:
  - 0.0.0.0:9200:9200
volumes:
  - elasticsearch_data:/usr/share/elasticsearch/data
environment:
  TZ: "Europe/Berlin"
  ES_JAVA_OPTS: -Xms512m -Xmx512m
  xpack.security.enabled: "true"
restart: on-failure:5

Dockerfilel:
FROM docker.elastic.co/elasticsearch/elasticsearch:7.4.1

COPY --chown=elasticsearch:elasticsearch elasticsearch.yml /usr/share/elasticsearch/config/

USER root
RUN chmod -R 777 /usr/share/elasticsearch/data
USER elasticsearch

elasticsearch.yml:
http.cors.enabled: true
http.cors.allow-origin: "*"
network.host: 0.0.0.0
script.engine.groovy.inline.update: true
script.engine.groovy.inline.aggs: true
node.name: "BrewMaster"

```

---

<div class="post-metadata">

**Author:** ![Benni](https://avatars.discourse-cdn.com/v4/letter/b/c67d28/32.png) [@Benni](https://discuss.elastic.co/u/Benni)\
**Post date:** [October 29, 2019, 4:22pm UTC](https://discuss.elastic.co/t/failed-to-obtain-node-locks-tried-usr-share-elasticsearch-data-with-lock-id-0/205706/2 "2019-10-29T16:22:27Z")

</div>

The permissions about the folders could still be quite interesting.

> drwxrwxr-x 1 elasticsearch root 4096 Oct 22 17:22 .  
> drwxr-xr-x 1 root root 4096 Oct 22 17:22 ..  
> -rw-r--r-- 1 elasticsearch root 18 Oct 30 2018 .bash\_logout  
> -rw-r--r-- 1 elasticsearch root 193 Oct 30 2018 .bash\_profile  
> -rw-r--r-- 1 elasticsearch root 231 Oct 30 2018 .bashrc  
> -rw-r--r-- 1 elasticsearch root 13675 Oct 22 17:14 LICENSE.txt  
> -rw-r--r-- 1 elasticsearch root 523209 Oct 22 17:20 NOTICE.txt  
> -rw-r--r-- 1 elasticsearch root 8500 Oct 22 17:14 README.textile  
> drwxr-xr-x 2 elasticsearch root 4096 Oct 22 17:21 bin  
> drwxrwxr-x 1 elasticsearch root 4096 Oct 29 12:14 config  
> drwxrwxrwx 1 elasticsearch root 4096 Oct 22 17:21 data  
> drwxr-xr-x 1 elasticsearch root 4096 Oct 22 17:20 jdk  
> drwxr-xr-x 3 elasticsearch root 4096 Oct 22 17:20 lib  
> drwxrwxr-x 2 elasticsearch root 4096 Oct 22 17:20 logs  
> drwxr-xr-x 37 elasticsearch root 4096 Oct 22 17:20 modules  
> drwxr-xr-x 2 elasticsearch root 4096 Oct 22 17:20 plugins

---

<div class="post-metadata">

**Author:** ![Benni](https://avatars.discourse-cdn.com/v4/letter/b/c67d28/32.png) [@Benni](https://discuss.elastic.co/u/Benni)\
**Post date:** [October 30, 2019, 12:40pm UTC](https://discuss.elastic.co/t/failed-to-obtain-node-locks-tried-usr-share-elasticsearch-data-with-lock-id-0/205706/3 "2019-10-30T12:40:59Z")

</div>

I think I solved it at least partially. At least the output has changed. I set **node.max\_local\_storage\_nodes: 20**. Which was a good match for the error. Now there is a new problem.

> elasticsearch\_1 | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.  
> elasticsearch\_1 | [2019-10-30T13:33:02,560][WARN][o.e.b.JNANatives] [BrewMaster] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory  
> elasticsearch\_1 | [2019-10-30T13:33:02,575][WARN][o.e.b.JNANatives] [BrewMaster] This can result in part of the JVM being swapped out.  
> elasticsearch\_1 | [2019-10-30T13:33:02,576][WARN][o.e.b.JNANatives] [BrewMaster] Increase RLIMIT\_MEMLOCK, soft limit: 16777216, hard limit: 16777216  
> elasticsearch\_1 | [2019-10-30T13:33:02,577][WARN][o.e.b.JNANatives] [BrewMaster] These can be adjusted by modifying /etc/security/limits.conf, for example:  
> elasticsearch\_1 | # allow user 'elasticsearch' mlockall  
> elasticsearch\_1 | elasticsearch soft memlock unlimited  
> elasticsearch\_1 | elasticsearch hard memlock unlimited  
> elasticsearch\_1 | [2019-10-30T13:33:02,578][WARN][o.e.b.JNANatives] [BrewMaster] If you are logged in interactively, you will have to re-login for the new limits to take effect.  
> elasticsearch\_1 | [2019-10-30T13:33:03,535][WARN][o.e.b.ElasticsearchUncaughtExceptionHandler] [BrewMaster] uncaught exception in thread [main]  
> elasticsearch\_1 | org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes/1];  
> elasticsearch\_1 | at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.4.1.jar:7.4.1]  
> elasticsearch\_1 | at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.4.1.jar:7.4.1]  
> elasticsearch\_1 | at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.4.1.jar:7.4.1]  
> elasticsearch\_1 | at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.4.1.jar:7.4.1]  
> elasticsearch\_1 | at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.4.1.jar:7.4.1]  
> elasticsearch\_1 | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.4.1.jar:7.4.1]  
> elasticsearch\_1 | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.4.1.jar:7.4.1]  
> elasticsearch\_1 | Caused by: org.elasticsearch.ElasticsearchException: failed to bind service  
> elasticsearch\_1 | at org.elasticsearch.node.Node.(Node.java:614) ~[elasticsearch-7.4.1.jar:7.4.1]  
> elasticsearch\_1 | at org.elasticsearch.node.Node.(Node.java:255) ~[elasticsearch-7.4.1.jar:7.4.1]  
> elasticsearch\_1 | at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]  
> elasticsearch\_1 | at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]  
> elasticsearch\_1 | at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.1.jar:7.4.1]  
> elasticsearch\_1 | at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.1.jar:7.4.1]  
> elasticsearch\_1 | ... 6 more  
> elasticsearch\_1 | Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes/1  
> elasticsearch\_1 | at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]  
> elasticsearch\_1 | at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]  
> elasticsearch\_1 | at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]  
> elasticsearch\_1 | at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) ~[?:?]  
> elasticsearch\_1 | at java.nio.file.Files.createDirectory(Files.java:693) ~[?:?]  
> elasticsearch\_1 | at java.nio.file.Files.createAndCheckIsDirectory(Files.java:800) ~[?:?]  
> elasticsearch\_1 | at java.nio.file.Files.createDirectories(Files.java:786) ~[?:?]  
> elasticsearch\_1 | at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:272) ~[elasticsearch-7.4.1.jar:7.4.1]  
> elasticsearch\_1 | at org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:209) ~[elasticsearch-7.4.1.jar:7.4.1]  
> elasticsearch\_1 | at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:269) ~[elasticsearch-7.4.1.jar:7.4.1]  
> elasticsearch\_1 | at org.elasticsearch.node.Node.(Node.java:275) ~[elasticsearch-7.4.1.jar:7.4.1]  
> elasticsearch\_1 | at org.elasticsearch.node.Node.(Node.java:255) ~[elasticsearch-7.4.1.jar:7.4.1]  
> elasticsearch\_1 | at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]  
> elasticsearch\_1 | at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]  
> elasticsearch\_1 | at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.1.jar:7.4.1]  
> elasticsearch\_1 | at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.1.jar:7.4.1]  
> elasticsearch\_1 | ... 6 more

But as far as I can see, I have enough permissions for everything. I don't quite know why he says "AccessDeniedException[/usr/share/elasticsearch/data/nodes/1]".  
Help would be very nice

---

<div class="post-metadata">

**Author:** ![Benni](https://avatars.discourse-cdn.com/v4/letter/b/c67d28/32.png) [@Benni](https://discuss.elastic.co/u/Benni)\
**Post date:** [October 30, 2019, 2:02pm UTC](https://discuss.elastic.co/t/failed-to-obtain-node-locks-tried-usr-share-elasticsearch-data-with-lock-id-0/205706/4 "2019-10-30T14:02:53Z")

</div>

The next bug is fixed.  
I had activated the _bootstrap.memory\_lock_ in **docker-compose.yml** , I undone it and set "_ES\_HEAP\_SIZE: 1g_" and "_MAY\_LOCKED\_MEMORY: unlimited_".  
In **elasticsearch.yml** I set "_bootstrap.mlockall: true_".

> OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.  
> [2019-10-30T14:34:57,309][WARN][o.e.b.ElasticsearchUncaughtExceptionHandler] [BrewMaster] uncaught exception in thread [main]  
> org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes/1];  
> at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.4.1.jar:7.4.1]  
> at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.4.1.jar:7.4.1]  
> at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.4.1.jar:7.4.1]  
> Caused by: org.elasticsearch.ElasticsearchException: failed to bind service  
> at org.elasticsearch.node.Node.(Node.java:614) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.node.Node.(Node.java:255) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.1.jar:7.4.1]  
> ... 6 more  
> Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes/1  
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]  
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]  
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]  
> at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) ~[?:?]  
> at java.nio.file.Files.createDirectory(Files.java:693) ~[?:?]  
> at java.nio.file.Files.createAndCheckIsDirectory(Files.java:800) ~[?:?]  
> at java.nio.file.Files.createDirectories(Files.java:786) ~[?:?]  
> at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:272) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:209) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:269) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.node.Node.(Node.java:275) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.node.Node.(Node.java:255) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.1.jar:7.4.1]  
> ... 6 more

---

<div class="post-metadata">

**Author:** ![Magister](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@Magister](https://discuss.elastic.co/u/Magister)\
**Post date:** [October 31, 2019, 3:32am UTC](https://discuss.elastic.co/t/failed-to-obtain-node-locks-tried-usr-share-elasticsearch-data-with-lock-id-0/205706/5 "2019-10-31T03:32:58Z")

</div>

Hi, I met the same issue [https://github.com/elastic/elasticsearch/issues/48374](https://github.com/elastic/elasticsearch/issues/48374).

How did you find these bug? I want to reproduce this problem, but it does not appear anymore.

The first time I met this bug, I reboot machine and restart elasticsearch.  
After this, some nodes could not obtain node locks, I guess this might be a lucene bug...

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [October 31, 2019, 7:39am UTC](https://discuss.elastic.co/t/failed-to-obtain-node-locks-tried-usr-share-elasticsearch-data-with-lock-id-0/205706/6 "2019-10-31T07:39:36Z")

</div>

This is the problem:

> [@Benni](#):
>
> ```plaintext
> Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes/0/node.lock
> 
> ```
> 
> and
> 
> ```plaintext
> Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes/1
> 
> ```

`AccessDeniedException` here means that Elasticsearch is prevented from accessing these files by your operating system. This is nothing to do with settings like `bootstrap.memory_lock` or `bootstrap.mlockall`, it's a file permissions issue.

> [@Benni](#):
>
> I set **node.max\_local\_storage\_nodes: 20**

You should not set this setting. It is a little dangerous and is deprecated.

---

<div class="post-metadata">

**Author:** ![Benni](https://avatars.discourse-cdn.com/v4/letter/b/c67d28/32.png) [@Benni](https://discuss.elastic.co/u/Benni)\
**Post date:** [October 31, 2019, 8:17am UTC](https://discuss.elastic.co/t/failed-to-obtain-node-locks-tried-usr-share-elasticsearch-data-with-lock-id-0/205706/7 "2019-10-31T08:17:03Z")

</div>

> [@DavidTurner](#):
>
> This is the problem:
> 
> > [@Benni](#):
> >
> > ```plaintext
> > Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes/0/node.lock
> > 
> > ```
> > 
> > and
> > 
> > ```plaintext
> > Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes/1
> > 
> > ```
> 
> `AccessDeniedException` here means that Elasticsearch is prevented from accessing these files by your operating system. This is nothing to do with settings like `bootstrap.memory_lock` or `bootstrap.mlockall`, it's a file permissions issue.
> 
> > [@Benni](#):
> >
> > I set **node.max\_local\_storage\_nodes: 20**
> 
> You should not set this setting. It is a little dangerous and is deprecated.

Thanks for the answer.  
That the java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes/1 is not a problem from bootstrap.memory\_lock or bootstrap.mlockall was almost clear to me but it fixed other bugs.  
What I would like to know is:  
How can I solve this with AccessDeniedException? How can I give the container the permissions it needs?

I've tried these methods:

> #RUN chmod -R 777 /usr/share/elasticsearch/data/  
> #RUN chgrp 1000 /usr/share/elasticsearch/data/  
> #RUN chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data/

That didn't work, so I tried that:

> RUN mkdir -p config data logs  
> RUN chmod -R 0777 config data logs  
> #RUN chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data/  
> RUN mkdir -p /usr/share/elasticsearch/data/nodes/  
> RUN chown -R 1000:1000 /usr/share/elasticsearch/data

That doesn't work either. (My current status)  
The output remains unchanged.

> OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.  
> [2019-10-30T18:15:15,679][WARN][o.e.b.ElasticsearchUncaughtExceptionHandler] [BrewMaster] uncaught exception in thread [main]  
> org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes/1];  
> at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.4.1.jar:7.4.1]  
> at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.4.1.jar:7.4.1]  
> at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.4.1.jar:7.4.1]  
> Caused by: org.elasticsearch.ElasticsearchException: failed to bind service  
> at org.elasticsearch.node.Node.(Node.java:614) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.node.Node.(Node.java:255) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.1.jar:7.4.1]  
> ... 6 more  
> Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes/1  
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]  
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]  
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]  
> at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) ~[?:?]  
> at java.nio.file.Files.createDirectory(Files.java:693) ~[?:?]  
> at java.nio.file.Files.createAndCheckIsDirectory(Files.java:800) ~[?:?]  
> at java.nio.file.Files.createDirectories(Files.java:786) ~[?:?]  
> at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:272) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:209) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:269) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.node.Node.(Node.java:275) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.node.Node.(Node.java:255) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.1.jar:7.4.1]  
> at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.1.jar:7.4.1]  
> ... 6 more

So what would you generally suggest to me?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [October 31, 2019, 8:28am UTC](https://discuss.elastic.co/t/failed-to-obtain-node-locks-tried-usr-share-elasticsearch-data-with-lock-id-0/205706/8 "2019-10-31T08:28:57Z")

</div>

Your `docker-compose` file above is quite a long way from the one in the [manual](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#_notes_for_production_use_and_defaults). Can you reproduce the problem with the recommended configuration? The manual also contains [notes on file permissions](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#_notes_for_production_use_and_defaults).

---

<div class="post-metadata">

**Author:** ![Benni](https://avatars.discourse-cdn.com/v4/letter/b/c67d28/32.png) [@Benni](https://discuss.elastic.co/u/Benni)\
**Post date:** [October 31, 2019, 1:21pm UTC](https://discuss.elastic.co/t/failed-to-obtain-node-locks-tried-usr-share-elasticsearch-data-with-lock-id-0/205706/9 "2019-10-31T13:21:13Z")

</div>

I changed it after the manual and adjusted it a bit. This is how it looks now and the error still occurs. I'll show you a bit more of the configuration.  
I hope it helps to find the bug.  
docker-compose.yml

```
elasticsearch:
build:
  context: elasticsearch
ports:
  - 0.0.0.0:9200:9200
environment:
  - node.name=es01
  - discovery.seed_hosts=es02
  - cluster.initial_master_nodes=es01,es02
  - cluster.name=docker-cluster
  - bootstrap.memory_lock=true
  - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
  - node.max_local_storage_nodes=20
ulimits:
  memlock:
    soft: -1
    hard: -1
volumes:
  - elasticsearch_data:/usr/share/elasticsearch/data
ports:
  - 9200:9200
restart: on-failure:5
[....]
networks:
 default:
   driver: bridge
   ipam:
    driver: default
    config:
      - subnet: 172.18.99.0/24
        ip_range: 172.18.99.0/24
        gateway: 172.18.99.1
volumes:
  grafana_data:
    driver: local-persist
    driver_opts:
      mountpoint: /volumes/monitoring/grafana/data
  grafana_plugins:
  elasticsearch_data:
    driver: local-persist
    driver_opts:
       mountpoint: /volumes/monitoring/elasticsearch/data
  graphite_data:
    driver: local-persist
    driver_opts:
       mountpoint: /volumes/monitoring/graphite/data

```

elasticsearch Dockerfile

```
FROM docker.elastic.co/elasticsearch/elasticsearch:7.4.1

#USER root
#RUN chmod -R 777 /usr/share/elasticsearch/data/
#RUN chgrp 1000 /usr/share/elasticsearch/data/
#RUN chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data/
#USER elasticsearch

ENV PATH /usr/share/elasticsearch/bin:$PATH

#RUN groupadd -g 1000 elasticsearch && adduser -u 1000 -g 1000 -d /usr/share/elasticsearch elasticsearch

WORKDIR /usr/share/elasticsearch
RUN cd /opt && curl --retry 8 -s -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.4.1-linux-x86_64.tar.gz && cd -
RUN tar zxf /opt/elasticsearch-7.4.1-linux-x86_64.tar.gz --strip-components=1
RUN grep ES_DISTRIBUTION_TYPE=tar /usr/share/elasticsearch/bin/elasticsearch-env && sed -ie 's/ES_DISTRIBUTION_TYPE=tar/ES_DISTRIBUTION_TYPE=docker/' /usr/share/elasticsearch/bin/elasticsearch-env
RUN usermod -a -G 0 elasticsearch
RUN mkdir -p config data logs
RUN chmod -R 0777 config data logs

#RUN chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data/

RUN mkdir -p /usr/share/elasticsearch/data/nodes/
RUN chown -R 1000:1000 /usr/share/elasticsearch/data
RUN rm -rf /etc/security/limits.conf
#COPY --chown=elasticsearch:elasticsearch elasticsearch.yml /usr/share/elasticsearch/config/
COPY --chown=elasticsearch:elasticsearch limits.conf /etc/security/
#USER elasticsearch

```

By the way,  
when I deleted "node.max\_local\_storage\_nodes=20" the error:

> " Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max\_local\_storage\_nodes] (was [1])?"

was there again  
That's why I'm letting it go unless there's an alternate. : )

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [October 31, 2019, 1:45pm UTC](https://discuss.elastic.co/t/failed-to-obtain-node-locks-tried-usr-share-elasticsearch-data-with-lock-id-0/205706/10 "2019-10-31T13:45:01Z")

</div>

> [@Benni](#):
>
> when I deleted "node.max\_local\_storage\_nodes=20" the error:
> 
> > " Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max\_local\_storage\_nodes] (was [1])?"
> 
> was there again

That's just another consequence of the same file permission problems, right? In your earlier post you shared a stack trace showing that the root cause of this exception was also an `AccessDeniedException`:

> [@Benni](#):
>
> Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes/0/node.lock

There's something strange going on in your environment to do with file permissions (and as always Docker is certainly not making this any simpler). I think that I cannot help you debug this much more than that - this is more of a general sysadmin question than something specific to Elasticsearch.

---

<div class="post-metadata">

**Author:** ![Benni](https://avatars.discourse-cdn.com/v4/letter/b/c67d28/32.png) [@Benni](https://discuss.elastic.co/u/Benni)\
**Post date:** [November 1, 2019, 7:54am UTC](https://discuss.elastic.co/t/failed-to-obtain-node-locks-tried-usr-share-elasticsearch-data-with-lock-id-0/205706/11 "2019-11-01T07:54:44Z")

</div>

Okay, thanks for the help.  
I will now try to rebuild the system.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 29, 2019, 7:54am UTC](https://discuss.elastic.co/t/failed-to-obtain-node-locks-tried-usr-share-elasticsearch-data-with-lock-id-0/205706/12 "2019-11-29T07:54:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.