Repository verification exception

Elastic Stack Elasticsearch
1

Hi all
i have a 5 node cluster where 3 nodes are data nodes, 1 is master and 1 is client node
on this cluster when trying to create snapshot i am getting following error

{
  "error": {
    "root_cause": [
      {
        "type": "repository_verification_exception",
        "reason": "[sgpl_backup2] [[Uu67NQqXQVaT1maJ49apng, 'RemoteTransportException[[node72][192.168.1.72:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[sgpl_backup2] a file written by master to the store [/home/sgpl/elasticsearch_backup] cannot be accessed on the node [{node72}{Uu67NQqXQVaT1maJ49apng}{IQI8z2v7QzeohCBSFOZHRA}{192.168.1.72}{192.168.1.72:9300}{ml.enabled=true}]. This might indicate that the store [/home/sgpl/elasticsearch_backup] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node];'], [mHR3zxDqQCqDvGNBxUF2tw, 'RemoteTransportException[[node73][192.168.1.73:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[sgpl_backup2] a file written by master to the store [/home/sgpl/elasticsearch_backup] cannot be accessed on the node [{node73}{mHR3zxDqQCqDvGNBxUF2tw}{BwILk21wR4GlJaWDy-DHFg}{192.168.1.73}{192.168.1.73:9300}{ml.enabled=true}]. This might indicate that the store [/home/sgpl/elasticsearch_backup] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node];'], [XEgaW0CvQt-02VEczOmHoA, 'RemoteTransportException[[node71][192.168.1.71:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[sgpl_backup2] a file written by master to the store [/home/sgpl/elasticsearch_backup] cannot be accessed on the node [{node71}{XEgaW0CvQt-02VEczOmHoA}{WMgwMgp_QLae61ux_xPqtA}{192.168.1.71}{192.168.1.71:9300}{ml.enabled=true}]. This might indicate that the store [/home/sgpl/elasticsearch_backup] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node];']]"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[sgpl_backup2] [[Uu67NQqXQVaT1maJ49apng, 'RemoteTransportException[[node72][192.168.1.72:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[sgpl_backup2] a file written by master to the store [/home/sgpl/elasticsearch_backup] cannot be accessed on the node [{node72}{Uu67NQqXQVaT1maJ49apng}{IQI8z2v7QzeohCBSFOZHRA}{192.168.1.72}{192.168.1.72:9300}{ml.enabled=true}]. This might indicate that the store [/home/sgpl/elasticsearch_backup] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node];'], [mHR3zxDqQCqDvGNBxUF2tw, 'RemoteTransportException[[node73][192.168.1.73:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[sgpl_backup2] a file written by master to the store [/home/sgpl/elasticsearch_backup] cannot be accessed on the node [{node73}{mHR3zxDqQCqDvGNBxUF2tw}{BwILk21wR4GlJaWDy-DHFg}{192.168.1.73}{192.168.1.73:9300}{ml.enabled=true}]. This might indicate that the store [/home/sgpl/elasticsearch_backup] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node];'], [XEgaW0CvQt-02VEczOmHoA, 'RemoteTransportException[[node71][192.168.1.71:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[sgpl_backup2] a file written by master to the store [/home/sgpl/elasticsearch_backup] cannot be accessed on the node [{node71}{XEgaW0CvQt-02VEczOmHoA}{WMgwMgp_QLae61ux_xPqtA}{192.168.1.71}{192.168.1.71:9300}{ml.enabled=true}]. This might indicate that the store [/home/sgpl/elasticsearch_backup] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node];']]"
  },
  "status": 500
}

how can i resolve this error ?

Thank you

2

Home did you mount the shared dir on all nodes?

3

This is my repo path
path.repo: ["/home/sgpl/elasticsearch_backup"]

with the help of below command i have given all permission to that repo.path and where sgpl is user of elasticsearch
sudo chown -R sgpl:sgpl /home/sgpl/elasticsearch_backup/

4

Is this a shared filesystem that is available to ALL the nodes in the cluster at the same path?

5

Yes it is available in all DATA( 3 nodes ) and Master( 1 node ) nodes ,
but not in client node .

6

What kind of device is it? NFS volume?

1 Like
7 
[root@elasticsearch_masternode ~]# cat /etc/redhat-release 
CentOS Linux release 7.2.1511 (Core) 

[root@elasticsearch_masternode ~]# uname -a
Linux elasticsearch_masternode 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

and as of know i do not have much knowledge about NFS and how it (NFS) is a part of snapshot and recovery process

8

You need to have a shared volume, e.g. a NFS volume, behind the repository path that all nodes can access. This means that if node1 writes a file it will be visible by node 2 and node 3. A directory in the local file system will therefore not work, even if the path is identical on all machines.

9

would you suggest me how do i have a shared volume, e.g. a NFS volume, behind the repository path that all nodes can access.
if possible provide me some web links which helps me.

Thank you.

3 Likes
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.