# Access Denied Exception in elasticsearch v5.6

**URL:** <https://discuss.elastic.co/t/access-denied-exception-in-elasticsearch-v5-6/206276>\
**Category:** Elasticsearch\
**Created:** [November 3, 2019, 8:01am UTC](https://discuss.elastic.co/t/access-denied-exception-in-elasticsearch-v5-6/206276 "2019-11-03T08:01:44Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![amirhossein](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@amirhossein](https://discuss.elastic.co/u/amirhossein)\
**Post date:** [November 3, 2019, 8:01am UTC](https://discuss.elastic.co/t/access-denied-exception-in-elasticsearch-v5-6/206276/1 "2019-11-03T08:01:44Z")

</div>

Hi, i have a cluster with 3 nodes. elasticsearch UID is different and their GID is same, But when I get the information of the snapshot on the Kibana, all the processes failed, here is part of the logs of the snapshot:

`nested: ElasticsearchException[failed to create blob container]; nested: AccessDeniedException`

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [November 4, 2019, 10:14am UTC](https://discuss.elastic.co/t/access-denied-exception-in-elasticsearch-v5-6/206276/2 "2019-11-04T10:14:04Z")

</div>

can you share the full exception from the elasticsearch logs? Thank you!

---

<div class="post-metadata">

**Author:** ![amirhossein](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@amirhossein](https://discuss.elastic.co/u/amirhossein)\
**Post date:** [November 4, 2019, 10:27am UTC](https://discuss.elastic.co/t/access-denied-exception-in-elasticsearch-v5-6/206276/3 "2019-11-04T10:27:29Z")

</div>

Thank you for your reply, error is:

```
 "error": {
"root_cause": [
  {
    "type": "repository_verification_exception",
"reason": "[backup] [[ldGSpx-ZQe6fBmgwI_hrgQ, 'RemoteTransportException[[....184:9300][internal:admin/repository/verify]];
nested: RepositoryMissingException[[backup] missing];'], [MWHreh4FQLmMa_yzbjKthQ, 'RemoteTransportException[[...][....183:9300][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[backup] store location [/mnt/elasticsearch]
 is not accessible on the node [{...}{MWHreh4FQLmMa_yzbjKthQ}{jwAAK-D2RquuiohxDfLSmw}{....183}{....183:9300}]]; 
 nested: AccessDeniedException[/mnt/elasticsearch/tests-dDepnDzCIA/data-MWHrLmMa_yzbjKthQ.dat];']]"
  }
]
},
"status": 500

```

{...} is my host names or ip, [/mnt/elasticsearch] is **NAS storage**.

and this error:

`Caused by: java.nio.file.AccessDeniedException: /mnt/elasticsearch/backup/tests-6xN....W_KZQ/data-M.....dat`

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [November 4, 2019, 1:42pm UTC](https://discuss.elastic.co/t/access-denied-exception-in-elasticsearch-v5-6/206276/4 "2019-11-04T13:42:30Z")

</div>

Is `/mnt/elasticsearch` accessible on that host? Did you configure the `path.repo` setting?

---

<div class="post-metadata">

**Author:** ![amirhossein](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@amirhossein](https://discuss.elastic.co/u/amirhossein)\
**Post date:** [November 4, 2019, 1:51pm UTC](https://discuss.elastic.co/t/access-denied-exception-in-elasticsearch-v5-6/206276/5 "2019-11-04T13:51:30Z")

</div>

yes, yml file is configured, 2 nodes (..184 and ..185) can write on /mnt/elasticsearch/ but ..183 cant create file on this repo.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [November 4, 2019, 1:52pm UTC](https://discuss.elastic.co/t/access-denied-exception-in-elasticsearch-v5-6/206276/6 "2019-11-04T13:52:11Z")

</div>

can you store a file there as the user running elasticsearch in a shell?

---

<div class="post-metadata">

**Author:** ![amirhossein](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@amirhossein](https://discuss.elastic.co/u/amirhossein)\
**Post date:** [November 4, 2019, 1:55pm UTC](https://discuss.elastic.co/t/access-denied-exception-in-elasticsearch-v5-6/206276/7 "2019-11-04T13:55:25Z")

</div>

yes, elasticsearch user can read and write files in a shell, but elasticsearch service can't!!!

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [November 4, 2019, 3:21pm UTC](https://discuss.elastic.co/t/access-denied-exception-in-elasticsearch-v5-6/206276/8 "2019-11-04T15:21:03Z")

</div>

just to verify: how exactly did you test this?

---

<div class="post-metadata">

**Author:** ![amirhossein](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@amirhossein](https://discuss.elastic.co/u/amirhossein)\
**Post date:** [November 4, 2019, 3:30pm UTC](https://discuss.elastic.co/t/access-denied-exception-in-elasticsearch-v5-6/206276/9 "2019-11-04T15:30:18Z")

</div>

with "sudo -H -u elasticsearch bash -c 'vim test.txt'" command on /mnt/elasticsearch/

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [November 4, 2019, 4:19pm UTC](https://discuss.elastic.co/t/access-denied-exception-in-elasticsearch-v5-6/206276/10 "2019-11-04T16:19:22Z")

</div>

have you tried to write/read the resource mentioned in the exception?

---

<div class="post-metadata">

**Author:** ![amirhossein](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@amirhossein](https://discuss.elastic.co/u/amirhossein)\
**Post date:** [November 4, 2019, 6:31pm UTC](https://discuss.elastic.co/t/access-denied-exception-in-elasticsearch-v5-6/206276/11 "2019-11-04T18:31:09Z")

</div>

Sorry I don't understand what you mean

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [November 5, 2019, 9:17am UTC](https://discuss.elastic.co/t/access-denied-exception-in-elasticsearch-v5-6/206276/12 "2019-11-05T09:17:21Z")

</div>

The exception refers to a file it tries to write at `/mnt/elasticsearch/tests-dDepnDzCIA/data-MWHrLmMa_yzbjKthQ.dat` - can you try to create that resource via `touch` on the command line?

---

<div class="post-metadata">

**Author:** ![amirhossein](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@amirhossein](https://discuss.elastic.co/u/amirhossein)\
**Post date:** [November 5, 2019, 9:47am UTC](https://discuss.elastic.co/t/access-denied-exception-in-elasticsearch-v5-6/206276/13 "2019-11-05T09:47:52Z")

</div>

Thanks for your answer  
yes i can do it, when i force node 185 to create file on /mnt/elasticsearch/backup1/ it is done, and when i force node 183 to create file on /mnt/elasticsearch/backup/ it is done too,  
but when node 185 is going to create /mnt/elasticsearch/backup/ then node 183 can not read/write files on this dir.  
note: primary gid related to elasticsearch on 183 is not equal to node 185, but there is another group on node 183 which is same as primary gid of elasticsearch on 185.  
in all nodes UMask of elasticsearch proccess is set to: 0002

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [November 5, 2019, 9:57am UTC](https://discuss.elastic.co/t/access-denied-exception-in-elasticsearch-v5-6/206276/14 "2019-11-05T09:57:00Z")

</div>

Thanks for testing. This means, that your system is not set up the same on all nodes. I suppose you need to fix your network storage setup then (or your network file system), as it does not look like an Elasticsearch issue to me. Mapping the GIDs sounds like a good start, but it's hard to tell without knowing the exact setup.

---

<div class="post-metadata">

**Author:** ![amirhossein](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@amirhossein](https://discuss.elastic.co/u/amirhossein)\
**Post date:** [November 5, 2019, 10:33am UTC](https://discuss.elastic.co/t/access-denied-exception-in-elasticsearch-v5-6/206276/15 "2019-11-05T10:33:11Z")

</div>

Thanks for your support

---

<div class="post-metadata">

**Author:** ![amirhossein](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@amirhossein](https://discuss.elastic.co/u/amirhossein)\
**Post date:** [November 11, 2019, 12:44pm UTC](https://discuss.elastic.co/t/access-denied-exception-in-elasticsearch-v5-6/206276/16 "2019-11-11T12:44:02Z")

</div>

if i change primary GID of elasticsearch user, What will be happened?

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [November 20, 2019, 3:20pm UTC](https://discuss.elastic.co/t/access-denied-exception-in-elasticsearch-v5-6/206276/17 "2019-11-20T15:20:36Z")

</div>

I don't know, as there might be a uid/gid mapping in your network file system that I am not aware off. Also, you may need to change some permissions of your elasticsearch installation (purging the package might make sense in this case). Best would be to test this out in a staging system.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 18, 2019, 3:24pm UTC](https://discuss.elastic.co/t/access-denied-exception-in-elasticsearch-v5-6/206276/18 "2019-12-18T15:24:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
