Elasticsearch Startup Error with NFS repo for backup

Elastic Stack Elasticsearch
1

Elasticsearch doesn't start with repo as NFS mount-point.
Elasticsearch Version:

[root@ip-10-92-3-133 ~]# curl -XGET localhost:9200
{
  "name" : "ip-10-92-3-133.ap-south-1.compute.internal",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "RsgjpoxYR7eCh5W2vccn5Q",
  "version" : {
    "number" : "7.9.1",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "083627f112ba94dffc1232e8b42b73492789ef91",
    "build_date" : "2020-09-01T21:22:21.964974Z",
    "build_snapshot" : false,
    "lucene_version" : "8.6.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Mount-point details:
xx.xx.xxx.xxx:/nfs_IPCL 350G 22G 329G 7% /root/elasticsearch-backup

Permission of the mount-point directory
drwxrwxrwx. 2 elasticsearch elasticsearch 43 Feb 22 08:18 elasticsearch-backup

Entry in Elasticsearch.yml file
path.repo: ["/root/elasticsearch-backup"]

Error in logfile:

[2022-02-22T08:10:39,891][ERROR][o.e.b.Bootstrap          ] [ip-10-92-3-133.ap-south-1.compute.internal] Exception
java.lang.IllegalStateException: Unable to access 'path.repo' (/root/elasticsearch-backup)
        at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:70) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:313) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:252) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.Security.configure(Security.java:121) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:222) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) [elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) [elasticsearch-cli-7.9.1.jar:7.9.1]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.9.1.jar:7.9.1]
Caused by: java.nio.file.AccessDeniedException: /root/elasticsearch-backup
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) ~[?:?]
        at java.nio.file.Files.createDirectory(Files.java:694) ~[?:?]
        at java.nio.file.Files.createAndCheckIsDirectory(Files.java:801) ~[?:?]
        at java.nio.file.Files.createDirectories(Files.java:787) ~[?:?]
        at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:389) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:68) ~[elasticsearch-7.9.1.jar:7.9.1]
        ... 12 more
[2022-02-22T08:10:39,908][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [ip-10-92-3-133.ap-south-1.compute.internal] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Unable to access 'path.repo' (/root/elasticsearch-backup)
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.9.1.jar:7.9.1]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.9.1.jar:7.9.1]
Caused by: java.lang.IllegalStateException: Unable to access 'path.repo' (/root/elasticsearch-backup)
        at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:70) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:313) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:252) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.Security.configure(Security.java:121) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:222) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.1.jar:7.9.1]
        ... 6 more
Caused by: java.nio.file.AccessDeniedException: /root/elasticsearch-backup
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) ~[?:?]
        at java.nio.file.Files.createDirectory(Files.java:694) ~[?:?]
        at java.nio.file.Files.createAndCheckIsDirectory(Files.java:801) ~[?:?]
        at java.nio.file.Files.createDirectories(Files.java:787) ~[?:?]
        at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:389) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:68) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:313) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:252) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.Security.configure(Security.java:121) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:222) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.1.jar:7.9.1]
        ... 6 more

Any help in resolving this would be a great help!

2

Should these two not match?

3

Thanks @Christian_Dahlqvist but I somehow mixed up the yaml file after you pointed out I corrected my question still getting the same error.

4

Does the elasticsearch has permissions to the /root directory as well?

You shared that it has permissions to /root/elasticsearch-backup, but it also needs permission to the /root directory, which is a bad practice.

I would not recommend changing permissions to the /root directory as you should not use the /root directory for anything else that is not executed by the root user.

It would be better to use another path, like /opt/elastic-backup or even create an /elastic-backup with full permissions to the elasticsearch user.

1 Like
5

Quick note that 7.9 is now EOL, and you should really upgrade :slight_smile:

1 Like
6

Thanks @leandrojmp!!!
the permission was the issue created another directory in / and viola

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.