# Snapshots on NFS volume - AccessDenied Exception

**URL:** <https://discuss.elastic.co/t/snapshots-on-nfs-volume-accessdenied-exception/221855>\
**Category:** Elasticsearch\
**Created:** [March 3, 2020, 10:40am UTC](https://discuss.elastic.co/t/snapshots-on-nfs-volume-accessdenied-exception/221855 "2020-03-03T10:40:31Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![MattJones](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattjones/32/55706_2.png) [@MattJones](https://discuss.elastic.co/u/MattJones)\
**Post date:** [March 3, 2020, 10:40am UTC](https://discuss.elastic.co/t/snapshots-on-nfs-volume-accessdenied-exception/221855/1 "2020-03-03T10:40:32Z")

</div>

Hi,

We're getting an intermittent AccessDeniedException when creating snapshots on our cluster. As far as I can see all permissions are correct and config is the same across all servers.

ES Version: `7.6.0`

The `/home/shopfiles` directory is a shared NFS mount across all servers.

The exception:

```auto
RepositoryException[[shopfiles] cannot create blob store]; nested: AccessControlException[access denied ("java.io.FilePermission" "/home/shopfiles/private/elasticsearch" "read")];
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:424)
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.shardContainer(BlobStoreRepository.java:987)
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.shardContainer(BlobStoreRepository.java:983)
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.snapshotShard(BlobStoreRepository.java:1468)
	at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:340)
	at org.elasticsearch.snapshots.SnapshotShardsService.lambda$startNewShards$1(SnapshotShardsService.java:286)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:830)
	Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/home/shopfiles/private/elasticsearch" "read")
	at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
	at java.base/java.security.AccessController.checkPermission(AccessController.java:1036)
	at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:408)
	at java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:747)
	at java.base/sun.nio.fs.UnixPath.checkRead(UnixPath.java:818)
	at java.base/sun.nio.fs.UnixFileSystemProvider.checkAccess(UnixFileSystemProvider.java:295)
	at java.base/java.nio.file.Files.createDirectories(Files.java:765)
	at org.elasticsearch.common.blobstore.fs.FsBlobStore.<init>(FsBlobStore.java:46)
	at org.elasticsearch.repositories.fs.FsRepository.createBlobStore(FsRepository.java:119)
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:420)
	... 9 more

```

Config on all servers:

```auto
root@es1 /e/elasticsearch# cat /etc/elasticsearch/elasticsearch.yml | grep path.repo
path.repo: ["/home/shopfiles/private/elasticsearch/snapshots"]

```

Listing directory mentioned in exception as Elasticsearch user:

```auto
root@es1 /e/elasticsearch# sudo -u elasticsearch -H ls -al /home/shopfiles/private/elasticsearch
total 32
drwxrwxr-x 8 myuser myuser 4096 Dec 19 2018 .
drwxr-xr-x 5 myuser myuser 4096 Jan 31 21:49 ..
drwxrwxr-x 3 elasticsearch elasticsearch 4096 Mar 3 10:15 snapshots

```

I have checked the UID / GID of Elasticsearch on all nodes, they are the same.

```auto
root@es5 /e/elasticsearch# cat /etc/passwd | grep elasticsearch
elasticsearch:x:113:116::/nonexistent:/bin/false

```

Any ideas?

---

<div class="post-metadata">

**Author:** ![MattJones](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattjones/32/55706_2.png) [@MattJones](https://discuss.elastic.co/u/MattJones)\
**Post date:** [March 9, 2020, 11:15am UTC](https://discuss.elastic.co/t/snapshots-on-nfs-volume-accessdenied-exception/221855/2 "2020-03-09T11:15:22Z")

</div>

As an update to this, I couldn't make any progress so I tried deleting the repository from Elasticsearch.

When attempting to recreate it with the same settings, I had the same error. For the sake of thoroughness I tried `chmod`ing the parent directory to `777`. After this I was able to recreate the repository.

I then changed the perms back to `775` and it still worked!? Something funky going on internally here perhaps?

## Before:

```auto
drwxrwxr-x 8 myuser myuser 4096 Dec 19 2018 ./
drwxr-xr-x 5 myuser myuser 4096 Jan 31 21:49 ../
drwxrwxr-x 3 elasticsearch elasticsearch 4096 Mar 3 10:15 snapshots

```

```auto
PUT http://xxxxxxx:9200/_snapshot/shopfiles
{
    "type": "fs",
    "settings": {
        "location": "/home/shopfiles/private/elasticsearch/snapshots",
        "compress": true
    }
}

```

```auto
{
  "error": {
    "root_cause": [
      {
        "type": "repository_exception",
        "reason": "[shopfiles] cannot create blob store"
      }
    ],
    "type": "repository_exception",
    "reason": "[shopfiles] cannot create blob store",
    "caused_by": {
      "type": "security_exception",
      "reason": "access denied (\"java.io.FilePermission\" \"/home/shopfiles/private/elasticsearch\" \"read\")"
    }
  },
  "status": 500
}

```

## 777 Test:

```auto
chmod 777 /home/shopfiles/private/elasticsearch

```

```auto
drwxrwxrwx 8 myuser myuser 4096 Dec 19 2018 ./
drwxr-xr-x 5 myuser myuser 4096 Jan 31 21:49 ../
drwxrwxr-x 3 elasticsearch elasticsearch 4096 Mar 9 11:10 snapshots/

```

```auto
PUT http://xxxxxxx:9200/_snapshot/shopfiles
{
    "type": "fs",
    "settings": {
        "location": "/home/shopfiles/private/elasticsearch/snapshots",
        "compress": true
    }
}

```

```auto
{
  "acknowledged": true
}

```

I thought this was really odd, so I deleted the repository again and gave it another go with the right permissions:

## After:

```auto
chmod 775 /home/shopfiles/private/elasticsearch

```

```auto
drwxrwxrwx 8 myuser myuser 4096 Dec 19 2018 ./
drwxr-xr-x 5 myuser myuser 4096 Jan 31 21:49 ../
drwxrwxr-x 3 elasticsearch elasticsearch 4096 Mar 9 11:10 snapshots/

```

```auto
PUT http://xxxxxxx:9200/_snapshot/shopfiles
{
    "type": "fs",
    "settings": {
        "location": "/home/shopfiles/private/elasticsearch/snapshots",
        "compress": true
    }
}

```

```auto
{
  "acknowledged": true
}

```

What!??

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 6, 2020, 11:15am UTC](https://discuss.elastic.co/t/snapshots-on-nfs-volume-accessdenied-exception/221855/3 "2020-04-06T11:15:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.