Error when creating a Snapshot Repository

[Zamiell (James)]

Greetings!

I'm following the Snapshot and Restore guide listed here:
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html

I'm running a fairly large ElasticSearch cluster on 6.4.2. All my nodes run CentOS. I've set up an NFS mount for every node in the cluster, as explained in the guide.

However, upon creating/verifying a new snapshot repository, I get an error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
[snip]

and

nested: RepositoryMissingException[[my_backup] missing];']

and

nested: RepositoryVerificationException[[my_backup] store location [/mnt/elasticsearch-backup/kibana_backup] is not accessible on the node [{es-18}

and

nested: AccessDeniedException[/mnt/elasticsearch-backup/kibana_backup/tests-F2_sNB9QSJyRjrAOxigw2g/data-kEVEhOgBRni_hTeGVGnJsQ.dat];']

and then more of the same.

So at first this looks like a permissions issue, but I don't think that it is. Both /mnt/elasticsearch-backup and /mnt/elasticsearch-backup/kibana_backup have 777 permissions. Furthermore, I've verified that all of them are connected properly with the mount command, and when I create a file on one node, it does appear in the directory of another node, so the NFS mount appears to be functioning correctly.

I did some Google searching and I wasn't able to find anything. Anyone have any tips?

Thank you,

[DavidTurner]

It's better to share the whole stack trace of the exception and the whole error message - even though they can be quite long, they also include useful information.

I'm going to guess that the Elasticsearch process is not running as the same numerical UID or GID on each node, and your NFS configuration does not compensate for this, meaning that when one node creates a subdirectory of the repository it is not possible for the other nodes to modify it.

[Zamiell (James)]

Thank you @DavidTurner. I'll ensure that all ElasticSearch users are on the same UID/GID, and then try again.

I didn't want to post the full stack trace out of fear of leaking something sensitive.