Hello. Yesterday one of our data nodes left our cluster and we've been unable to get it to rejoin. I'm still able to SSH into the physical server and noticed an error in the elasticsearch logs that led me to discover that the /media/Data/elasticsearch folder and all of its contents are missing. I checked the elasticsearch.yml file and confirmed it's set to that path.
Path to directory containing configuration (this file and logging.yml):
path.data: /media/Data/elasticsearch
I also checked a working data node and confirmed that the /media/Data/elasticsearch folder and its contents were still there.
Any ideas on what would have caused this and what the remedy would be? Would an elasticsearch reinstall be all that's needed for a fix?
Including the error from the missing data node Elasticsearch logs below:
[2022-10-17T12:49:17,328][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [data8] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Unable to access 'path.data' (/media/Data/elasticsearch)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.16.3.jar:7.16.3]
at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.16.3.jar:7.16.3]
Caused by: java.lang.IllegalStateException: Unable to access 'path.data' (/media/Data/elasticsearch)
at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:66) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:226) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:168) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Security.configure(Security.java:115) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:229) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.16.3.jar:7.16.3]
... 6 more
Caused by: java.nio.file.AccessDeniedException: /media/Data/elasticsearch
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:398) ~[?:?]
at java.nio.file.Files.createDirectory(Files.java:700) ~[?:?]
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:807) ~[?:?]
at java.nio.file.Files.createDirectories(Files.java:793) ~[?:?]
at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:318) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:64) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:226) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:168) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Security.configure(Security.java:115) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:229) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.16.3.jar:7.16.3]
... 6 more