Unable to register NFS for snapshot

Elastic Stack Elasticsearch
1

Hi guys,

I'm unable to do the following cURL command:

curl -XPUT http://localhost:9200/_snapshot/backup -H 'Content-Type: application/json' -d '
{
    "type": "fs",
    "settings": {
        "location": "/mnt/NFS/NFS/logs",
        "compress": true
    }
}
'

The NFS is mounted at /mnt/NFS and the logs directory was created by the cURL command on all nodes. However, it presented me with this error:

{
  "error": {
    "root_cause": [
      {
        "type": "repository_verification_exception",
        "reason": "[backup] [[CbSAF_DDT869p9o4x9zFhg, 'RemoteTransportException[[logs-data-1][10.130.52.12:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[backup] store location [/mnt/NFS/NFS/logs] is not accessible on the node [{logs-data-1}{CbSAF_DDT869p9o4x9zFhg}{8si9G_SkSdqjpPbLhPqqdA}{10.130.52.12}{10.130.52.12:9300}{ml.machine_memory=8352534528, xpack.installed=true, ml.max_open_jobs=20}]]; nested: AccessDeniedException[/mnt/NFS/NFS/logs/tests-ZwoKJ3FNT8SCokXRnjfsRg/data-CbSAF_DDT869p9o4x9zFhg.dat];'], [Q75cM9JEQJSC4JofvAbNSA, 'RemoteTransportException[[logs-data-2][10.130.52.13:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[backup] store location [/mnt/NFS/NFS/logs] is not accessible on the node [{logs-data-2}{Q75cM9JEQJSC4JofvAbNSA}{2441G-fPTKa_wOFQMBhLXQ}{10.130.52.13}{10.130.52.13:9300}{ml.machine_memory=8353415168, xpack.installed=true, ml.max_open_jobs=20}]]; nested: AccessDeniedException[/mnt/NFS/NFS/logs/tests-ZwoKJ3FNT8SCokXRnjfsRg/data-Q75cM9JEQJSC4JofvAbNSA.dat];']]"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[backup] [[CbSAF_DDT869p9o4x9zFhg, 'RemoteTransportException[[logs-data-1][10.130.52.12:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[backup] store location [/mnt/NFS/NFS/logs] is not accessible on the node [{logs-data-1}{CbSAF_DDT869p9o4x9zFhg}{8si9G_SkSdqjpPbLhPqqdA}{10.130.52.12}{10.130.52.12:9300}{ml.machine_memory=8352534528, xpack.installed=true, ml.max_open_jobs=20}]]; nested: AccessDeniedException[/mnt/NFS/NFS/logs/tests-ZwoKJ3FNT8SCokXRnjfsRg/data-CbSAF_DDT869p9o4x9zFhg.dat];'], [Q75cM9JEQJSC4JofvAbNSA, 'RemoteTransportException[[logs-data-2][10.130.52.13:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[backup] store location [/mnt/NFS/NFS/logs] is not accessible on the node [{logs-data-2}{Q75cM9JEQJSC4JofvAbNSA}{2441G-fPTKa_wOFQMBhLXQ}{10.130.52.13}{10.130.52.13:9300}{ml.machine_memory=8353415168, xpack.installed=true, ml.max_open_jobs=20}]]; nested: AccessDeniedException[/mnt/NFS/NFS/logs/tests-ZwoKJ3FNT8SCokXRnjfsRg/data-Q75cM9JEQJSC4JofvAbNSA.dat];']]"
  },
  "status": 500
}

I also tried to first change the logs directory owner to elasticsearch:elasticsearch and was able to create file in it. However, once I ran the cURL command, the permissions changed and I was no longer able to write into it.

Could anyone tell me what I am doing wrong here? Thanks!

1 Like
2

NFS permission issues are often because the Elasticsearch process is not running as the same numerical UID or GID on each node, without compensating for this in the NFS configuration, meaning that when one node creates a subdirectory of the repository it is not possible for the other nodes to modify it. That is the first thing I would check.

3

Hi @DavidTurner,

Thanks! That is indeed the problem. I used usermod -u [uid] elasticsearch and groupmod -g [gid] elasticsearch to change the UID and GID. I did need to stop the cluster, re-chown directories and files that Elasticsearch uses and restart the cluster but that fixed it.

1 Like
4

@aqiank I have also changed the user id of the elasticsearch and the folder /var/lib/elasticsearch. But I am unable start my elasticsearch . Please tell me the solution

5

Hi @Vineeth_Ananthula1,

I also chown /etc/default/elasticsearch, /etc/elasticsearch/ and /var/log/elasticsearch.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.