*fixed* Repository Verification Exception on registering an repository

Elastic Stack Elasticsearch
1

Hi there,

I have a master-only node and 3 Data-Nodes in a Cluster running elasticsearch 5.1.1 and I want to make snapshots of some indexes. In every node, I set "path.repo: ["/opt/elasticsearch/archive"]" in the elasticsearch.yml. Now I try to register an repository using curl. Every time I do this, I get an repository_verification_exception with the reason:

RemoteTransportException[[data-1][xxx.xxx.xxx.xxx:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[archive] a file written by master to the store [/opt/elasticsearch/archive] cannot be accessed on the node XY

The folders are present and ownership is given to the elastic user, running the server. Checking on the servers, every node except the master node has a tests-XYZ file in the archive folder. A ls -ld archive/ check gives this for the master node: drwxr-xr-x 2 elastic elastic 4096 May 30 13:00 archive/

Any idea what I'm doing wrong?

Thanks so far

2

A small update from my side.

I made fresh installs of 5.4 instances on every server.
As I said on every node except the client/ingest one, I set:
path.repo: ["/opt/elasticsearch/snapshots"]

I created that folder, chowned it to the elastic user and even did chmod 777 on it.

The problem remains, when I try to register an snapshot repository. In every snapshots folder of the data nodes, a test folder is created with an hash, except the master one. And as result I just get the error from the first post (a 500 RepositoryVerificationException).

The registration looks like this out of cerebro, although I tried it with curl too:

_snapshot/snapshots
{
    "type": "fs",
    "settings": {
	"compress": false,
	"location": "/opt/elasticsearch/snapshots"
    }
}

My intention is to rename an specific index and store it on a specified node through a index-template and the routing.allocation. Reason behind that is archiving an old index on a slower server. I tried it with the reindex API but that was way to slow.

edit
I used the installation via tar if that matters

3

Got the problem: the network storage wasn't bind properly. The nodes could all write in their directory but of course wouldn't share their data. So the master thought the data-nodes weren't able to write into their folders.

That brought up another small question though. During restoring, I wanted the indices to be stored in specified archive-nodes. I did the shard/index allocation via index templates and filtered for the _name field to prevent storing normal data on archive nodes. That however means that the restored index, although now renamed to fit the archive index template, still had the index-allocation exclusion. I know that there is a way give index properties which should be ignored during the restore, but that don't seem to work with the index-allocation. So I had to overwrite it instead, which worked and I added an index-allocation include to only store in archive node (which seems to have lower priority to the exclude).

Anyway, is there another way to not restore the index-allocation settings of the index during the snapshot or restore and instead supply a new one? Or is overwriting the only way? I'd rather only have one include setting instead of one include and one exclude.

2 Likes
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.