Snapshot: repository_verification_exception

[rojin]

Hi! I am trying to do a snapshot for my cluster including 2 nodes. I have a shared file on both of my nodes. node-1 as an NFS server and node-2 as the client. if I add a file to the specified shared directory which is /var/nfsshare, I can see it on the other node. There's a directory in nfsshare called es-backup and I have granted permission to elasticsearch user on each directory on both nodes.
Here's my fstab content:

[...]
192.168.xx.xxx:/var/nfsshare/es-backup /var/nfs/es-backup-ovh nfs4 rw,_netdev,tcp 0 0

Here's the command to create a snapshot:

   PUT /_snapshot/test
    {
      "type": "fs",
      "settings": {
        "location": "es-backup",
        "compress": true
      }
    }

But when I want to create a snapshot on kibana dev tools:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[test] [[VNjwVdvSTqSzW5Ggxn3k0A, 'RemoteTransportException[[node-2][192.168.xx.xxx:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[test] store location [/var/nfsshare/es-backup] is not accessible on the node [{node-2}{VNjwVdvSTqSzW5Ggxn3k0A}{o7aZPvy4RvW1nkJ6Bn7k_A}{192.168.xx.xxx}{192.168.xx.xxx:9300}{dilmrt}{ml.machine_memory=6087491584, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]]; nested: AccessDeniedException[/var/nfsshare/es-backup/tests-XNfastCkTee1h5Pz_vf2uA/data-VNjwVdvSTqSzW5Ggxn3k0A.dat];']]"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[test] [[VNjwVdvSTqSzW5Ggxn3k0A, 'RemoteTransportException[[node-2][192.168.xx.xxx:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[test] store location [/var/nfsshare/es-backup] is not accessible on the node [{node-2}{VNjwVdvSTqSzW5Ggxn3k0A}{o7aZPvy4RvW1nkJ6Bn7k_A}{192.168.xx.xxx}{192.168.xx.xxx:9300}{dilmrt}{ml.machine_memory=6087491584, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]]; nested: AccessDeniedException[/var/nfsshare/es-backup/tests-XNfastCkTee1h5Pz_vf2uA/data-VNjwVdvSTqSzW5Ggxn3k0A.dat];']]"
  },
  "status" : 500
}

192.168.yy.yyy is where my NFS server's located (node-1) and 192.168.xx.xxx is where my client NFS is. Xpack is enabled on both nodes as well. I'd be so grateful if anyone could help me with this.

[DavidTurner]

@rojin

nested: AccessDeniedException

On NFS, this is usually bad permissions. See the docs for details:

IMPORTANT: By default, a network file system (NFS) uses user IDs (UIDs) and group IDs (GIDs) to match accounts across nodes. If your shared file system is an NFS and your nodes don’t use the same UIDs and GIDs, update your NFS configuration to account for this.

[rojin]

Thank you for the reply. How can I include UID and GID in NFS configuration? Thanks in advance.

[DavidTurner]

Sorry, you'll need to find an NFS expert for help with that. I know it's technically possible to run different machines with mismatched UIDs/GIDs, but I don't know the details.

[rojin]

Thank you for your time. I'll ask it.