Elasticsearch 7.3.0 - AccessDeniedException: /usr/share/elasticsearch/data/nodes

Hello,

I deployed elasticsearch on IBM IKS cluster but I see some java exception. Please let me know whether this is something with a specific version or different. The same version works fine on GKE.

k version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:08:12Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.9+IKS", GitCommit:"858a7bdfc1b3c9dd7a4a035e563a13eb87587de4", GitTreeState:"clean", BuildDate:"2019-08-07T11:03:12Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}


k get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                 STORAGECLASS   REASON   AGE
pvc-0dc00b99-c122-11e9-ab9e-62dde2ad7daf   30Gi       RWO            Delete           Bound    default/elasticsearch-master-elasticsearch-master-0   default                 12m
pvc-0dc43e92-c122-11e9-ab9e-62dde2ad7daf   30Gi       RWO            Delete           Bound    default/elasticsearch-master-elasticsearch-master-1   default                 13m
pvc-0dc806a3-c122-11e9-ab9e-62dde2ad7daf   30Gi       RWO            Delete           Bound    default/elasticsearch-master-elasticsearch-master-2   default                 12m


k get pvc
NAME                                          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
elasticsearch-master-elasticsearch-master-0   Bound    pvc-0dc00b99-c122-11e9-ab9e-62dde2ad7daf   30Gi       RWO            default        15m
elasticsearch-master-elasticsearch-master-1   Bound    pvc-0dc43e92-c122-11e9-ab9e-62dde2ad7daf   30Gi       RWO            default        15m
elasticsearch-master-elasticsearch-master-2   Bound    pvc-0dc806a3-c122-11e9-ab9e-62dde2ad7daf   30Gi       RWO            default        15m


k logs -f elasticsearch-master-0
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
{"type": "server", "timestamp": "2019-08-17T19:15:33,933+0000", "level": "WARN", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "uncaught exception in thread [main]" , 
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.3.0.jar:7.3.0]",
"at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.3.0.jar:7.3.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.3.0.jar:7.3.0]",
"Caused by: org.elasticsearch.ElasticsearchException: failed to bind service",
"at org.elasticsearch.node.Node.<init>(Node.java:617) ~[elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.3.0.jar:7.3.0]",
"... 6 more",
"Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes",
"at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]",
"at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) ~[?:?]",
"at java.nio.file.Files.createDirectory(Files.java:692) ~[?:?]",
"at java.nio.file.Files.createAndCheckIsDirectory(Files.java:799) ~[?:?]",
"at java.nio.file.Files.createDirectories(Files.java:785) ~[?:?]",
"at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:272) ~[elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:209) ~[elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:269) ~[elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.3.0.jar:7.3.0]",
"... 6 more"] }

can you share the configuration file, including the different file paths you have configured in the yaml file?

Hi @spinscale

The issue is that IBM IKS doesn't allow non-root user to mount the volume. (https://cloud.ibm.com/docs/containers?topic=containers-cs_troubleshoot_storage#file_app_failures)

In the following helm chart https://hub.helm.sh/charts/elastic/elasticsearch

there is no such section

chown elasticsearch:elasticsearch /usr/share/elasticsearch/data

Better example (but a bit more cumbersome) is https://github.com/helm/charts/blob/a1050da95e0dcb70151adb0376bc1cf9d785485c/stable/elasticsearch/templates/master-statefulset.yaml#L106

I wonder if anyone could fix

chown

for https://hub.helm.sh/charts/elastic/elasticsearch

Can you please open an issue at https://github.com/elastic/helm-charts describing your issue - thanks a lot!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.