I am encountering an issue while attempting to verify a snapshot repository in Elasticsearch. The repository is named old-indices. Below is the relevant log entry from the Elasticsearch cluster:

abdelkriem · December 31, 2024, 12:08pm

[2024-12-31T17:06:04,282] [WARN ] [r.suppressed ] [Elastic-Master-1] path: /_snapshot/old-indices/_verify, params: {repository=old-indices}, status: 500org.elasticsearch.repositories.RepositoryVerificationException: [old-indices] [[4Se7RLnxSSy1_7Ngi4XisA, 'org.elasticsearch.transport.RemoteTransportexception: [node-1][10.10.24.153:9300] [internal:admin/repository/verify] ']]

abdelkriem · December 31, 2024, 12:20pm

DavidTurner · December 31, 2024, 12:39pm

Hi @abdelkriem and welcome!

Please don't post images of text, just post the text itself. I can barely read the message you've shared, but it looks like it's this one right?

github.com

elastic/elasticsearch/blob/47be542459ee024801fc2fd8bea5e8cc5dea2e0e/server/src/main/java/org/elasticsearch/repositories/blobstore/BlobStoreRepository.java#L3828-L3840


      
          throw new RepositoryVerificationException(
              metadata.name(),
              "a file written by master to the store ["
                  + blobStore()
                  + "] cannot be accessed on the node ["
                  + localNode
                  + "]. "
                  + "This might indicate that the store ["
                  + blobStore()
                  + "] is not shared between this node and the master node or "
                  + "that permissions on the store don't allow reading files written by the master node",
              e
          );

If so, that's exactly the problem.

abdelkriem · December 31, 2024, 12:52pm

Thanks for your response, yes this is the problem:
[2024-12-31T17:09:26,113] [WARN ][o.e.r.V.TransportAction ] [node-1] [old-indices] failed to verify repository
org.elasticsearch.repositories.RepositoryVerificationException: [old-indices] a file written by master to the store [/home//snapshot/old-indices] cannot be accessed on the node [{node-1}{4Se7RLnxSSy1_7Ngi4XisA

{vSipvIAIQHSUFpA68jW6UQ} {node-1}{! :9300}{cdfhilmrstw}{8.15.0}{7000099-8512000} {ml.machine_memory=16429281280, ml.allocated_processors=8, ml.allocated_processors_double=8.0, ml.max_jvm_size=8216641536
ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0}]. This might indicate that the store [/home//snapshot/old-indices] is not shared between this node and the master node or that permissions on

the store don't allow reading files written by the master node

Christian_Dahlqvist · December 31, 2024, 1:55pm

All master and data nodes need to have read and write access to the same shared repository. Mounting separate volumes in the same place per node will not work as Elasticsearch will verify the repository by having one node write a file and another one read the same.

abdelkriem · January 5, 2025, 8:34am

This implies that the shared repository can only be mounted on a single node at a time.

Christian_Dahlqvist · January 5, 2025, 8:40am

No, it does not. Why would that be the case?

abdelkriem · January 6, 2025, 8:42am

I have built a three-node cluster, with all roles assigned to each of these nodes. I am now looking to take a snapshot of the indices. Based on the official documentation, my understanding is that I need to create a folder on all three nodes and set the path.repo parameter in the elasticsearch.yml file to ensure it is consistent across all nodes.

However, I recently came across guidance suggesting that the folder only needs to be created on one of the nodes. Could you please clarify which approach is correct? Specifically:

Should the snapshot repository folder be created on all nodes in the cluster, or is it sufficient to create it on just one node?
If it is required on all nodes, does the folder path need to be identical across them?

Your clarification on this matter will be greatly appreciated. Thank you in advance for your support!

Christian_Dahlqvist · January 6, 2025, 8:45am

Both of these are incorrect. You need to create a shared storage volume, e.g. NFS, and mount this on the same path on all master and data nodes and all nodes need to be able to both read and write to this single shared volume.

Topic		Replies	Views
Repository verification exception Elasticsearch	9	11340	July 27, 2017
Getting error while creating snapshot Elasticsearch	3	5373	July 5, 2017
Elasticsearch - Snapshot having unassigned shards and Repository verification exception Elasticsearch	2	1521	February 10, 2019
Repository_verification_exception Elasticsearch snapshot-and-restore	2	390	July 24, 2023
fixed Repository Verification Exception on registering an repository Elasticsearch	3	5912	June 30, 2017

I am encountering an issue while attempting to verify a snapshot repository in Elasticsearch. The repository is named old-indices. Below is the relevant log entry from the Elasticsearch cluster:

Related topics