Three-node Elasticsearch Cluster with docker-compose failing with stacktrace--["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes

[kommineni24 (r)]

Hi,

I am trying to spin up a three node Elasticsearch cluster with a docker-compose file. While spinning up the cluster only one Elasticsearch container is coming up whereas other two containers on different nodes are failing due to AccessDenied Exception.
I have given the same local directory permissions on all the three nodes.

Logs:

es02    | {"type": "server", "timestamp": "2020-12-14T16:41:01,887Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "es-docker-cluster", "node.name": "xxxxxxxxxx", "message": "uncaught exception in thread [main]",
es02    | "stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];",
es02    | "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.10.0.jar:7.10.0]",
es02    | "at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.10.0.jar:7.10.0]",
es02    | "at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.10.0.jar:7.10.0]",
es02    | "at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.10.0.jar:7.10.0]",
es02    | "at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.10.0.jar:7.10.0]",
es02    | "at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.10.0.jar:7.10.0]",
es02    | "at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.10.0.jar:7.10.0]",
es02    | "Caused by: org.elasticsearch.ElasticsearchException: failed to bind service",
es02    | "at org.elasticsearch.node.Node.<init>(Node.java:729) ~[elasticsearch-7.10.0.jar:7.10.0]",
es02    | "at org.elasticsearch.node.Node.<init>(Node.java:289) ~[elasticsearch-7.10.0.jar:7.10.0]",
es02    | "at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.10.0.jar:7.10.0]",
es02    | "at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.10.0.jar:7.10.0]",
es02    | "at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.10.0.jar:7.10.0]",
es02    | "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.10.0.jar:7.10.0]",
es02    | "... 6 more",
es02    | "Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes",
es02    | "at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]",
es02    | "at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]",
es02    | "at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]",
es02    | "at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388) ~[?:?]",
es02    | "at java.nio.file.Files.createDirectory(Files.java:694) ~[?:?]",
es02    | "at java.nio.file.Files.createAndCheckIsDirectory(Files.java:801) ~[?:?]",
es02    | "at java.nio.file.Files.createDirectories(Files.java:787) ~[?:?]",
es02    | "at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:275) ~[elasticsearch-7.10.0.jar:7.10.0]",
es02    | "at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:212) ~[elasticsearch-7.10.0.jar:7.10.0]",
es02    | "at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:272) ~[elasticsearch-7.10.0.jar:7.10.0]",
es02    | "at org.elasticsearch.node.Node.<init>(Node.java:362) ~[elasticsearch-7.10.0.jar:7.10.0]",
es02    | "at org.elasticsearch.node.Node.<init>(Node.java:289) ~[elasticsearch-7.10.0.jar:7.10.0]",
es02    | "at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.10.0.jar:7.10.0]",
es02    | "at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.10.0.jar:7.10.0]",
es02    | "at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.10.0.jar:7.10.0]",
es02    | "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.10.0.jar:7.10.0]",
es02    | "... 6 more"] }
es02    | uncaught exception in thread [main]
es02    | ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];
es02    | Likely root cause: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes
es02    |       at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
es02    |       at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
es02    |       at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
es02    |       at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
es02    |       at java.base/java.nio.file.Files.createDirectory(Files.java:694)
es02    |       at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:801)
es02    |       at java.base/java.nio.file.Files.createDirectories(Files.java:787)
es02    |       at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:275)
es02    |       at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:212)
es02    |       at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:272)
es02    |       at org.elasticsearch.node.Node.<init>(Node.java:362)
es02    |       at org.elasticsearch.node.Node.<init>(Node.java:289)
es02    |       at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227)
es02    |       at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227)
es02    |       at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393)
es02    |       at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
es02    |       at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
es02    |       at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
es02    |       at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
es02    |       at org.elasticsearch.cli.Command.main(Command.java:90)
es02    |       at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
es02    |       at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
es02    | For complete error details, refer to the log at /usr/share/elasticsearch/logs/es-docker-cluster.log
es02 exited with code 1

Listing permissions of my local mount directories.
On Node 1:
Running my Compose from this node 1.
drwxrwxrwx 2 elasticsearch elasticsearch 6 Dec 2 15:25 es_data01
Container is coming up without any issues

On Node 2:
drwxrwxrwx 2 elasticsearch elasticsearch 6 Dec 2 15:25 es_data02
Container is not coming up with Accessdenied exception

On Node 3:
drwxrwxrwx 2 elasticsearch elasticsearch 6 Dec 2 15:25 es_data03
Container is not coming up with Accessdenied exception

Attaching my composePreformatted text file.

    > version: '2.2'
>     services:
>       es01:
>         image: docker.elastic.co/elasticsearch/elasticsearch:7.10.0
>         container_name: es01
>         environment:
>           - node.name=node01
>           - discovery.seed_hosts=node02, node03
>           - cluster.initial_master_nodes=node01,node02, node03
>           - cluster.name=es-docker-cluster
>           - node.master=true
>           - node.data=true
>           - bootstrap.memory_lock=true
>           - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
>         ulimits:
>           memlock:
>             soft: -1
>             hard: -1
>         volumes:
>            #- data01:/usr/share/elasticsearch/data
>            - /es_data01:/usr/share/elasticsearch/data
>         ports:
>           - 9200:9200
>         networks:
>           - elastic
>       es02:
>         image: docker.elastic.co/elasticsearch/elasticsearch:7.10.0
>         container_name: es02
>         environment:
>           - node.name=node02
>           - discovery.seed_hosts=node01, node03
>           - cluster.initial_master_nodes=node01,node02, node03
>           - cluster.name=es-docker-cluster
>           - node.master=true
>           - node.data=true
>           - bootstrap.memory_lock=true
>           - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
>         ulimits:
>           memlock:
>             soft: -1
>             hard: -1
>         volumes:
>           #- data02:/usr/share/elasticsearch/data
>           - /es_data02:/usr/share/elasticsearch/data
>         networks:
>           - elastic
>       es03:
>         image: docker.elastic.co/elasticsearch/elasticsearch:7.10.0
>         container_name: es03
>         environment:
>           - node.name=node03
>           - discovery.seed_hosts=node01,node02
>           - cluster.initial_master_nodes=node01,node02, node03
>           - cluster.name=es-docker-cluster
>           - node.master=true
>           - node.data=true
>           - bootstrap.memory_lock=true
>           - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
>         ulimits:
>           memlock:
>             soft: -1
>             hard: -1
>         volumes:
>           #- data03:/usr/share/elasticsearch/data
>           - /es_data03:/usr/share/elasticsearch/data
>         networks:
>           - elastic
> 
>     volumes:
>       es_data01:
>         driver: local
>       es_data02:
>         driver: local
>       es_data03:
>         driver: local
> 
>     networks:
>       elastic:
>         driver: bridge

[vincenbr (Vincent Bréhin)]

Hi, what do you mean by nodes here ?

other two containers on different nodes

your docker-compose setup is inteded to run 3 instances of containers (elasticsearch "nodes") on the same machine (system "node").
node.name setting in your compose file has a meaning only inside the internal docker-network of your local machine. With this setup, you cannot span an elasticsearch cluster across several hosts. Is it what you're intending to achieve ?

[kommineni24 (r)]

Hi Vincenbr,

Thanks for your reply.
Yeah, I am trying spin-up cluster across three different nodes(VM'S).

[vincenbr (Vincent Bréhin)]

OK , in this case docker-compose is not really the right way to go.
for just spinning up your 3 nodes (vm's) es cluster setup, you can get inspiration in this post where the author starts 5 es-docker-nodes on 5 VMs (look at the 5 different IPs): https://xyzcoder.github.io/2020/07/22/how-to-deploy-an-elastic-search-cluster-consisting-of-multiple-hosts-using-es-docker-image.html
then maybe have a look on https://www.portainer.io/ to manage the containers in a more practical way.

[kommineni24 (r)]

So the compose file in documentation will only work with single-node?

Install Elasticsearch with Docker | Elasticsearch Reference [7.10] | Elastic

What will be the best way to spin up the cluster? I am trying to automate the cluster deploy so thought going with compose where I can predefine all the configurations needed.

[kommineni24 (r)]

@vincenbr
Any suggestions please?

[vincenbr (Vincent Bréhin)]

Hi tp,
I confirm that docker-compose is a tool which is essentially about deploying on a single host (https://docs.docker.com/compose/#common-use-cases) .
I agree that elastic doc could highlight this point more clearly, and upgrade its docker-compose file example to a newer version.
Idea (that I haven't tried) : upgrade your docker-compose file to a newer version (3+) and deploy it on a docker swarm (sort of a docker multi-host cluster) as in https://docs.docker.com/engine/swarm/stack-deploy/