500 - Internal Server : ErrorRepository_verification_exception, nested: RepositoryVerificationException[[backup] a file written by master to the store [C:\\Updates\\Backups\\ElasticSearch] cannot be accessed on the node

Elastic Stack Elasticsearch
1

I have a cluster of 5 nodes(3master,2 data), i have created directory in each node and included its path as path.repo in all of its yml's. then when i take snapshot its giving this error.
command i used
PUT _snapshot/backup
{
"type": "fs",
"settings": {
"compress" : true,
"location": "C:\Updates\Backups\ElasticSearch\"
}
}

{
"error" : {
"root_cause" : [
{
"type" : "repository_verification_exception",
"reason" : "[backup] [[s7sT_BywRuqjQZVcRKgTRA, 'RemoteTransportException[[ICA-DB4][192.168.100.214:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[backup] a file written by master to the store [C:\Updates\Backups\ElasticSearch] cannot be accessed on the node [{ICA-DB4}{s7sT_BywRuqjQZVcRKgTRA}{0_FdA3OlSueQo2QVU81yiw}{192.168.100.214}{192.168.100.214:9300}{cdhilrstw}{ml.machine_memory=8588910592, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]. This might indicate that the store [C:\Updates\Backups\ElasticSearch] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node]; nested: NoSuchFileException[C:\Updates\Backups\ElasticSearch\tests-eYIkJTw1TH6ECKpoynbMZA\master.dat];'], [TBQKi5SeQZ2rkXCflDV0LQ, 'RemoteTransportException[[ICA-DB3][192.168.100.213:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[backup] a file written by master to the store [C:\Updates\Backups\ElasticSearch] cannot be accessed on the node [{ICA-DB3}{TBQKi5SeQZ2rkXCflDV0LQ}{B1J75sCyQq-eQUuahZNCNA}{192.168.100.213}{192.168.100.213:9300}{cdhilmrstw}{ml.machine_memory=8588910592, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]. This might indicate that the store [C:\Updates\Backups\ElasticSearch] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node]; nested: NoSuchFileException[C:\Updates\Backups\ElasticSearch\tests-eYIkJTw1TH6ECKpoynbMZA\master.dat];'], [2J1nm1GcRP6ofx3uYOorcA, 'RemoteTransportException[[ICA-DB5][192.168.100.215:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[backup] a file written by master to the store [C:\Updates\Backups\ElasticSearch] cannot be accessed on the node [{ICA-DB5}{2J1nm1GcRP6ofx3uYOorcA}{T6Sj6FUpQBCxSJGIMf4eow}{192.168.100.215}{192.168.100.215:9300}{cdhilrstw}{ml.machine_memory=8588910592, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]. This might indicate that the store [C:\Updates\Backups\ElasticSearch] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node]; nested: NoSuchFileException[C:\Updates\Backups\ElasticSearch\tests-eYIkJTw1TH6ECKpoynbMZA\master.dat];'], [KYo2-qPbSDCfs9S59sB5Eg, 'RemoteTransportException[[ICA-DB1][192.168.100.211:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[backup] a file written by master to the store [C:\Updates\Backups\ElasticSearch] cannot be accessed on the node [{ICA-DB1}{KYo2-qPbSDCfs9S59sB5Eg}{gRudinalT-GgFD107fSHhQ}{192.168.100.211}{192.168.100.211:9300}{cdhilmrstw}{ml.machine_memory=8588910592, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]. This might indicate that the store [C:\Updates\Backups\ElasticSearch] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node]; nested: NoSuchFileException[C:\Updates\Backups\ElasticSearch\tests-eYIkJTw1TH6ECKpoynbMZA\master.dat];']]"
}
],
"type" : "repository_verification_exception",
"reason" : "[backup] [[s7sT_BywRuqjQZVcRKgTRA, 'RemoteTransportException[[ICA-DB4][192.168.100.214:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[backup] a file written by master to the store [C:\Updates\Backups\ElasticSearch] cannot be accessed on the node [{ICA-DB4}{s7sT_BywRuqjQZVcRKgTRA}{0_FdA3OlSueQo2QVU81yiw}{192.168.100.214}{192.168.100.214:9300}{cdhilrstw}{ml.machine_memory=8588910592, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]. This might indicate that the store [C:\Updates\Backups\ElasticSearch] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node]; nested: NoSuchFileException[C:\Updates\Backups\ElasticSearch\tests-eYIkJTw1TH6ECKpoynbMZA\master.dat];'], [TBQKi5SeQZ2rkXCflDV0LQ, 'RemoteTransportException[[ICA-DB3][192.168.100.213:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[backup] a file written by master to the store [C:\Updates\Backups\ElasticSearch] cannot be accessed on the node [{ICA-DB3}{TBQKi5SeQZ2rkXCflDV0LQ}{B1J75sCyQq-eQUuahZNCNA}{192.168.100.213}{192.168.100.213:9300}{cdhilmrstw}{ml.machine_memory=8588910592, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]. This might indicate that the store [C:\Updates\Backups\ElasticSearch] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node]; nested: NoSuchFileException[C:\Updates\Backups\ElasticSearch\tests-eYIkJTw1TH6ECKpoynbMZA\master.dat];'], [2J1nm1GcRP6ofx3uYOorcA, 'RemoteTransportException[[ICA-DB5][192.168.100.215:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[backup] a file written by master to the store [C:\Updates\Backups\ElasticSearch] cannot be accessed on the node [{ICA-DB5}{2J1nm1GcRP6ofx3uYOorcA}{T6Sj6FUpQBCxSJGIMf4eow}{192.168.100.215}{192.168.100.215:9300}{cdhilrstw}{ml.machine_memory=8588910592, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]. This might indicate that the store [C:\Updates\Backups\ElasticSearch] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node]; nested: NoSuchFileException[C:\Updates\Backups\ElasticSearch\tests-eYIkJTw1TH6ECKpoynbMZA\master.dat];'], [KYo2-qPbSDCfs9S59sB5Eg, 'RemoteTransportException[[ICA-DB1][192.168.100.211:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[backup] a file written by master to the store [C:\Updates\Backups\ElasticSearch] cannot be accessed on the node [{ICA-DB1}{KYo2-qPbSDCfs9S59sB5Eg}{gRudinalT-GgFD107fSHhQ}{192.168.100.211}{192.168.100.211:9300}{cdhilmrstw}{ml.machine_memory=8588910592, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]. This might indicate that the store [C:\Updates\Backups\ElasticSearch] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node]; nested: NoSuchFileException[C:\Updates\Backups\ElasticSearch\tests-eYIkJTw1TH6ECKpoynbMZA\master.dat];']]"
},
"status" : 500
}

2

If you want to create a shared filesystem snapshot this requires a shared filesystem, e.g. NFS, mounted on the same path across all nodes in the cluster. You can not back up to individual directores on the nodes even if the path is the same.

1 Like
3

But when i use shared file system, I am able to run Elasticsearch only by batch file using PowerShell. but when I run Elasticsearch as a service it gets stop suddenly.
I am confused when we include path.repo in yml and give the path of directory present in shared file system and run Elasticsearch as service it gets stop. but when i run as powershell its working fine

4

Why? You use the shared storage for the snapshot repository only, not the data path. I do not see how the type of storage used for the snapshot repository would affect how you start or run Elasticsearch. If it fails to start, please show the logs.

5

yes , I am using share folder only for repository and the data is stored on same local node

6

Thanks for your quick response

7 
this are the logs and i run elastic search as a windows service
[2022-08-12T15:09:22,644][INFO ][o.e.n.Node ] [A2_MMJ] closed
[2022-08-12T15:09:27,507][ERROR][o.e.b.Bootstrap ] [A2_MMJ] Exception
java.lang.IllegalStateException: Unable to access 'path.repo' (X:\Backup)
at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:66) ~[elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:242) ~[elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:168) ~[elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.bootstrap.Security.configure(Security.java:115) ~[elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:229) ~[elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) [elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) [elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) [elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) [elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) [elasticsearch-cli-7.16.2.jar:7.16.2]
at org.elasticsearch.cli.Command.main(Command.java:77) [elasticsearch-cli-7.16.2.jar:7.16.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) [elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) [elasticsearch-7.16.2.jar:7.16.2]
Caused by: java.nio.file.FileSystemException: X:\Backup: Unable to determine if root directory exists
at java.nio.file.Files.createDirectories(Files.java:782) ~[?:?]
at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:318) ~[elasticsearch-7.16.2.jar:7.16.2]
at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:64) ~[elasticsearch-7.16.2.jar:7.16.2]
... 12 more
[2022-08-12T15:09:27,523][WARN ][stderr ] [A2_MMJ] java.lang.IllegalStateException: Unable to access 'path.repo' (X:\Backup)
[2022-08-12T15:09:27,523][WARN ][stderr ] [A2_MMJ] Likely root cause: java.nio.file.FileSystemException: X:\Backup: Unable to determine if root directory exists
[2022-08-12T15:09:27,523][WARN ][stderr ] [A2_MMJ] at java.base/java.nio.file.Files.createDirectories(Files.java:782)
[2022-08-12T15:09:27,523][WARN ][stderr ] [A2_MMJ] at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:318)
[2022-08-12T15:09:27,523][WARN ][stderr ] [A2_MMJ] at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:64)
[2022-08-12T15:09:27,523][WARN ][stderr ] [A2_MMJ] at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:242)
[2022-08-12T15:09:27,523][WARN ][stderr ] [A2_MMJ] at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:168)
[2022-08-12T15:09:27,523][WARN ][stderr ] [A2_MMJ] at org.elasticsearch.bootstrap.Security.configure(Security.java:115)
[2022-08-12T15:09:27,523][WARN ][stderr ] [A2_MMJ] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:229)
[2022-08-12T15:09:27,523][WARN ][stderr ] [A2_MMJ] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434)
[2022-08-12T15:09:27,523][WARN ][stderr ] [A2_MMJ] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166)
[2022-08-12T15:09:27,523][WARN ][stderr ] [A2_MMJ] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157)
[2022-08-12T15:09:27,523][WARN ][stderr ] [A2_MMJ] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77)
[2022-08-12T15:09:27,523][WARN ][stderr ] [A2_MMJ] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)
[2022-08-12T15:09:27,523][WARN ][stderr ] [A2_MMJ] at org.elasticsearch.cli.Command.main(Command.java:77)
[2022-08-12T15:09:27,523][WARN ][stderr ] [A2_MMJ] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122)
[2022-08-12T15:09:27,523][WARN ][stderr ] [A2_MMJ] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)
[2022-08-12T15:09:27,523][WARN ][stderr ] [A2_MMJ] For complete error details, refer to the log at C:\Elastic\Elasticsearch\logs\elasticsearch.log
8

Seems to be a permissions issue on the backup storage.

9

But the thing is, when I run it as elasticsearh.bat file its working fine. This error happens when i run Elasticsearch as a widows service.
Can you please also suggest how can i take the restore on a new Elasticsearch node from a snapshot that was stored on the shared folder.

10

Does it run as different users depending on how you start it? I do not run Elasticsearch on Windows, so will unfortunately not be able to help with this issue.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.