Failed to verify all nodes have repository access

Elastic Stack Elasticsearch
1

hello, I have a problem regarding backup using curator, I use version 6.3.2 of Elasticsearch and curator in version 5.7.6 , backups are stored in a google gsc bucket with all permissions but this error always appears
Failed to complete action: snapshot. <class 'curator.exceptions.ActionError'>: Failed to verify all nodes have repository access: --- Got a 500 response from Elasticsearch. Error message: repository_verification_exception
when creating the snapshots, could you help me?

2

There are usually good reasons that Elasticsearch should verify the snapshot repository. However, in older versions, like 6.3, not all nodes responded quickly enough to prevent this sort of message.

If you're completely certain there isn't something else going wrong with the repository, there's a way to skip the check:

3

thanks for your feedback,

I'm storing this in a google gcs bucket, I used this parameter but it still fails

4

generated this error only on one of the data nodes, the permissions on the bucket are the same as on the other nodes

  • missing]","node_id":"GOuzdEwtQ66KIuBpzqbXNg","status":"INTERNAL_SERVER_ERROR"},{"index":"8yace29ju3-metrics-202203","index_uuid":"8yace29ju3-metrics-202203","shard_id":5,"reason":"RepositoryMissingException[[backup_metrics_sensedia_prod] missing]","node_id":"GOuzdEwtQ66KIuBpzqbXNg","status":"INTERNAL_SERVER_ERROR"},{"index":"f3k3ojyjhl-metrics-202202","index_uuid":"f3k3ojyjhl-metrics-202202","shard_id":3,"reason
    Shards":{"total":366,"failed":45,"successful":321}
5

Do you have the API call you used to create the repository? It seems that Elasticsearch itself cannot complete the request.

If you dial up Curator's logging to DEBUG level, you will see the full API call made to start the snapshot. If you were to execute that API call using curl or some other direct REST method, you would get the same result. It appears that Curator is not the problem if you're getting INTERNAL_SERVER_ERROR type messages (which are coming directly from Elasticsearch).

Either the shards are missing/unavailable, or the node in question ("node_id":"GOuzdEwtQ66KIuBpzqbXNg") is unable to contact/reach the repository, or something worse, perhaps. I can't tell from just these messages, but that's just not a good sign.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.