Failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]

Elastic Stack Elasticsearch
1

Hello, I have already found many approaches for this error. Unfortunately none of them helped me to solve my problem. It seems to be an old friend of Elasticsearch.
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?

Blockquote
elasticsearch_1 | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
elasticsearch_1 | {"type": "server", "timestamp": "2019-10-29T15:57:33,739+01:00", "level": "WARN", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "elasticsearch", "node.name": "BrewMaster", "message": "uncaught exception in thread [main]",
elasticsearch_1 | "stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?",
.
.
.
elasticsearch_1 | "Caused by: java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data/nodes/0",
elasticsearch_1 | "at org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]",
elasticsearch_1 | "at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:269) ~[elasticsearch-7.4.1.jar:7.4.1]",
elasticsearch_1 | "at org.elasticsearch.node.Node.(Node.java:275) ~[elasticsearch-7.4.1.jar:7.4.1]",
elasticsearch_1 | "at org.elasticsearch.node.Node.(Node.java:255) ~[elasticsearch-7.4.1.jar:7.4.1]",
elasticsearch_1 | "at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]",
elasticsearch_1 | "at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]",
elasticsearch_1 | "at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.1.jar:7.4.1]",
elasticsearch_1 | "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.1.jar:7.4.1]",
elasticsearch_1 | "... 6 more",
elasticsearch_1 | "Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes/0/node.lock",
elasticsearch_1 | "at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]",
elasticsearch_1 | "at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]",
elasticsearch_1 | "at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]",
elasticsearch_1 | "at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:182) ~[?:?]",
elasticsearch_1 | "at java.nio.channels.FileChannel.open(FileChannel.java:292) ~[?:?]",
elasticsearch_1 | "at java.nio.channels.FileChannel.open(FileChannel.java:345) ~[?:?]",
.
.
.

This is the error I get and this is how my docker file, composite file and my .yml file looks like.

dockerfile-compose.yml:
elasticsearch:
build:
  context: elasticsearch
ports:
  - 0.0.0.0:9200:9200
volumes:
  - elasticsearch_data:/usr/share/elasticsearch/data
environment:
  TZ: "Europe/Berlin"
  ES_JAVA_OPTS: -Xms512m -Xmx512m
  xpack.security.enabled: "true"
restart: on-failure:5


Dockerfilel:
FROM docker.elastic.co/elasticsearch/elasticsearch:7.4.1

COPY --chown=elasticsearch:elasticsearch elasticsearch.yml /usr/share/elasticsearch/config/

USER root
RUN chmod -R 777 /usr/share/elasticsearch/data
USER elasticsearch

elasticsearch.yml:
http.cors.enabled: true
http.cors.allow-origin: "*"
network.host: 0.0.0.0
script.engine.groovy.inline.update: true
script.engine.groovy.inline.aggs: true
node.name: "BrewMaster"
2

The permissions about the folders could still be quite interesting.

drwxrwxr-x 1 elasticsearch root 4096 Oct 22 17:22 .
drwxr-xr-x 1 root root 4096 Oct 22 17:22 ..
-rw-r--r-- 1 elasticsearch root 18 Oct 30 2018 .bash_logout
-rw-r--r-- 1 elasticsearch root 193 Oct 30 2018 .bash_profile
-rw-r--r-- 1 elasticsearch root 231 Oct 30 2018 .bashrc
-rw-r--r-- 1 elasticsearch root 13675 Oct 22 17:14 LICENSE.txt
-rw-r--r-- 1 elasticsearch root 523209 Oct 22 17:20 NOTICE.txt
-rw-r--r-- 1 elasticsearch root 8500 Oct 22 17:14 README.textile
drwxr-xr-x 2 elasticsearch root 4096 Oct 22 17:21 bin
drwxrwxr-x 1 elasticsearch root 4096 Oct 29 12:14 config
drwxrwxrwx 1 elasticsearch root 4096 Oct 22 17:21 data
drwxr-xr-x 1 elasticsearch root 4096 Oct 22 17:20 jdk
drwxr-xr-x 3 elasticsearch root 4096 Oct 22 17:20 lib
drwxrwxr-x 2 elasticsearch root 4096 Oct 22 17:20 logs
drwxr-xr-x 37 elasticsearch root 4096 Oct 22 17:20 modules
drwxr-xr-x 2 elasticsearch root 4096 Oct 22 17:20 plugins

3

I think I solved it at least partially. At least the output has changed. I set node.max_local_storage_nodes: 20. Which was a good match for the error. Now there is a new problem.

elasticsearch_1 | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
elasticsearch_1 | [2019-10-30T13:33:02,560][WARN ][o.e.b.JNANatives ] [BrewMaster] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
elasticsearch_1 | [2019-10-30T13:33:02,575][WARN ][o.e.b.JNANatives ] [BrewMaster] This can result in part of the JVM being swapped out.
elasticsearch_1 | [2019-10-30T13:33:02,576][WARN ][o.e.b.JNANatives ] [BrewMaster] Increase RLIMIT_MEMLOCK, soft limit: 16777216, hard limit: 16777216
elasticsearch_1 | [2019-10-30T13:33:02,577][WARN ][o.e.b.JNANatives ] [BrewMaster] These can be adjusted by modifying /etc/security/limits.conf, for example:
elasticsearch_1 | # allow user 'elasticsearch' mlockall
elasticsearch_1 | elasticsearch soft memlock unlimited
elasticsearch_1 | elasticsearch hard memlock unlimited
elasticsearch_1 | [2019-10-30T13:33:02,578][WARN ][o.e.b.JNANatives ] [BrewMaster] If you are logged in interactively, you will have to re-login for the new limits to take effect.
elasticsearch_1 | [2019-10-30T13:33:03,535][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [BrewMaster] uncaught exception in thread [main]
elasticsearch_1 | org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes/1];
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.4.1.jar:7.4.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.4.1.jar:7.4.1]
elasticsearch_1 | at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.4.1.jar:7.4.1]
elasticsearch_1 | at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.4.1.jar:7.4.1]
elasticsearch_1 | at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.4.1.jar:7.4.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.4.1.jar:7.4.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.4.1.jar:7.4.1]
elasticsearch_1 | Caused by: org.elasticsearch.ElasticsearchException: failed to bind service
elasticsearch_1 | at org.elasticsearch.node.Node.(Node.java:614) ~[elasticsearch-7.4.1.jar:7.4.1]
elasticsearch_1 | at org.elasticsearch.node.Node.(Node.java:255) ~[elasticsearch-7.4.1.jar:7.4.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.1.jar:7.4.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.1.jar:7.4.1]
elasticsearch_1 | ... 6 more
elasticsearch_1 | Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes/1
elasticsearch_1 | at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
elasticsearch_1 | at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
elasticsearch_1 | at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
elasticsearch_1 | at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) ~[?:?]
elasticsearch_1 | at java.nio.file.Files.createDirectory(Files.java:693) ~[?:?]
elasticsearch_1 | at java.nio.file.Files.createAndCheckIsDirectory(Files.java:800) ~[?:?]
elasticsearch_1 | at java.nio.file.Files.createDirectories(Files.java:786) ~[?:?]
elasticsearch_1 | at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:272) ~[elasticsearch-7.4.1.jar:7.4.1]
elasticsearch_1 | at org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:209) ~[elasticsearch-7.4.1.jar:7.4.1]
elasticsearch_1 | at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:269) ~[elasticsearch-7.4.1.jar:7.4.1]
elasticsearch_1 | at org.elasticsearch.node.Node.(Node.java:275) ~[elasticsearch-7.4.1.jar:7.4.1]
elasticsearch_1 | at org.elasticsearch.node.Node.(Node.java:255) ~[elasticsearch-7.4.1.jar:7.4.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.1.jar:7.4.1]
elasticsearch_1 | at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.1.jar:7.4.1]
elasticsearch_1 | ... 6 more

But as far as I can see, I have enough permissions for everything. I don't quite know why he says "AccessDeniedException[/usr/share/elasticsearch/data/nodes/1]".
Help would be very nice

4

The next bug is fixed.
I had activated the bootstrap.memory_lock in docker-compose.yml, I undone it and set "ES_HEAP_SIZE: 1g" and "MAY_LOCKED_MEMORY: unlimited".
In elasticsearch.yml I set "bootstrap.mlockall: true".

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2019-10-30T14:34:57,309][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [BrewMaster] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes/1];
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.4.1.jar:7.4.1]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.4.1.jar:7.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.4.1.jar:7.4.1]
Caused by: org.elasticsearch.ElasticsearchException: failed to bind service
at org.elasticsearch.node.Node.(Node.java:614) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.node.Node.(Node.java:255) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.1.jar:7.4.1]
... 6 more
Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes/1
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) ~[?:?]
at java.nio.file.Files.createDirectory(Files.java:693) ~[?:?]
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:800) ~[?:?]
at java.nio.file.Files.createDirectories(Files.java:786) ~[?:?]
at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:272) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:209) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:269) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.node.Node.(Node.java:275) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.node.Node.(Node.java:255) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.1.jar:7.4.1]
... 6 more

5

Hi, I met the same issue https://github.com/elastic/elasticsearch/issues/48374.

How did you find these bug? I want to reproduce this problem, but it does not appear anymore.

The first time I met this bug, I reboot machine and restart elasticsearch.
After this, some nodes could not obtain node locks, I guess this might be a lucene bug...

6

This is the problem:

AccessDeniedException here means that Elasticsearch is prevented from accessing these files by your operating system. This is nothing to do with settings like bootstrap.memory_lock or bootstrap.mlockall, it's a file permissions issue.

You should not set this setting. It is a little dangerous and is deprecated.

7

Thanks for the answer.
That the java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes/1 is not a problem from bootstrap.memory_lock or bootstrap.mlockall was almost clear to me but it fixed other bugs.
What I would like to know is:
How can I solve this with AccessDeniedException? How can I give the container the permissions it needs?

I've tried these methods:

#RUN chmod -R 777 /usr/share/elasticsearch/data/
#RUN chgrp 1000 /usr/share/elasticsearch/data/
#RUN chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data/

That didn't work, so I tried that:

RUN mkdir -p config data logs
RUN chmod -R 0777 config data logs
#RUN chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data/
RUN mkdir -p /usr/share/elasticsearch/data/nodes/
RUN chown -R 1000:1000 /usr/share/elasticsearch/data

That doesn't work either. (My current status)
The output remains unchanged.

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2019-10-30T18:15:15,679][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [BrewMaster] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes/1];
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.4.1.jar:7.4.1]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.4.1.jar:7.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.4.1.jar:7.4.1]
Caused by: org.elasticsearch.ElasticsearchException: failed to bind service
at org.elasticsearch.node.Node.(Node.java:614) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.node.Node.(Node.java:255) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.1.jar:7.4.1]
... 6 more
Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes/1
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) ~[?:?]
at java.nio.file.Files.createDirectory(Files.java:693) ~[?:?]
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:800) ~[?:?]
at java.nio.file.Files.createDirectories(Files.java:786) ~[?:?]
at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:272) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:209) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:269) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.node.Node.(Node.java:275) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.node.Node.(Node.java:255) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.1.jar:7.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.1.jar:7.4.1]
... 6 more

So what would you generally suggest to me?

8

Your docker-compose file above is quite a long way from the one in the manual. Can you reproduce the problem with the recommended configuration? The manual also contains notes on file permissions.

9

I changed it after the manual and adjusted it a bit. This is how it looks now and the error still occurs. I'll show you a bit more of the configuration.
I hope it helps to find the bug.
docker-compose.yml

elasticsearch:
build:
  context: elasticsearch
ports:
  - 0.0.0.0:9200:9200
environment:
  - node.name=es01
  - discovery.seed_hosts=es02
  - cluster.initial_master_nodes=es01,es02
  - cluster.name=docker-cluster
  - bootstrap.memory_lock=true
  - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
  - node.max_local_storage_nodes=20
ulimits:
  memlock:
    soft: -1
    hard: -1
volumes:
  - elasticsearch_data:/usr/share/elasticsearch/data
ports:
  - 9200:9200
restart: on-failure:5
[....]
networks:
 default:
   driver: bridge
   ipam:
    driver: default
    config:
      - subnet: 172.18.99.0/24
        ip_range: 172.18.99.0/24
        gateway: 172.18.99.1
volumes:
  grafana_data:
    driver: local-persist
    driver_opts:
      mountpoint: /volumes/monitoring/grafana/data
  grafana_plugins:
  elasticsearch_data:
    driver: local-persist
    driver_opts:
       mountpoint: /volumes/monitoring/elasticsearch/data
  graphite_data:
    driver: local-persist
    driver_opts:
       mountpoint: /volumes/monitoring/graphite/data

elasticsearch Dockerfile

FROM docker.elastic.co/elasticsearch/elasticsearch:7.4.1

#USER root
#RUN chmod -R 777 /usr/share/elasticsearch/data/
#RUN chgrp 1000 /usr/share/elasticsearch/data/
#RUN chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data/
#USER elasticsearch

ENV PATH /usr/share/elasticsearch/bin:$PATH

#RUN groupadd -g 1000 elasticsearch &&     adduser -u 1000 -g 1000 -d /usr/share/elasticsearch   elasticsearch

WORKDIR /usr/share/elasticsearch
RUN cd /opt && curl --retry 8 -s -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.4.1-linux-x86_64.tar.gz && cd -
RUN tar zxf /opt/elasticsearch-7.4.1-linux-x86_64.tar.gz --strip-components=1
RUN grep ES_DISTRIBUTION_TYPE=tar /usr/share/elasticsearch/bin/elasticsearch-env     && sed -ie 's/ES_DISTRIBUTION_TYPE=tar/ES_DISTRIBUTION_TYPE=docker/' /usr/share/elasticsearch/bin/elasticsearch-env
RUN usermod -a -G 0 elasticsearch
RUN mkdir -p config data logs
RUN chmod -R 0777 config data logs

#RUN chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data/

RUN mkdir -p /usr/share/elasticsearch/data/nodes/
RUN chown -R 1000:1000 /usr/share/elasticsearch/data
RUN rm -rf /etc/security/limits.conf
#COPY --chown=elasticsearch:elasticsearch elasticsearch.yml /usr/share/elasticsearch/config/
COPY --chown=elasticsearch:elasticsearch limits.conf /etc/security/
#USER elasticsearch

By the way,
when I deleted "node.max_local_storage_nodes=20" the error:

" Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?"

was there again
That's why I'm letting it go unless there's an alternate. : )

10

That's just another consequence of the same file permission problems, right? In your earlier post you shared a stack trace showing that the root cause of this exception was also an AccessDeniedException:

There's something strange going on in your environment to do with file permissions (and as always Docker is certainly not making this any simpler). I think that I cannot help you debug this much more than that - this is more of a general sysadmin question than something specific to Elasticsearch.

11

Okay, thanks for the help.
I will now try to rebuild the system.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.