Access Denied Exception in elasticsearch v5.6

Elastic Stack Elasticsearch
1

Hi, i have a cluster with 3 nodes. elasticsearch UID is different and their GID is same, But when I get the information of the snapshot on the Kibana, all the processes failed, here is part of the logs of the snapshot:

nested: ElasticsearchException[failed to create blob container]; nested: AccessDeniedException

2

can you share the full exception from the elasticsearch logs? Thank you!

3

Thank you for your reply, error is:

 "error": {
"root_cause": [
  {
    "type": "repository_verification_exception",
"reason": "[backup] [[ldGSpx-ZQe6fBmgwI_hrgQ, 'RemoteTransportException[[....184:9300][internal:admin/repository/verify]];
nested: RepositoryMissingException[[backup] missing];'], [MWHreh4FQLmMa_yzbjKthQ, 'RemoteTransportException[[...][....183:9300][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[backup] store location [/mnt/elasticsearch]
 is not accessible on the node [{...}{MWHreh4FQLmMa_yzbjKthQ}{jwAAK-D2RquuiohxDfLSmw}{....183}{....183:9300}]]; 
 nested: AccessDeniedException[/mnt/elasticsearch/tests-dDepnDzCIA/data-MWHrLmMa_yzbjKthQ.dat];']]"
  }
]
},
"status": 500

{...} is my host names or ip, [/mnt/elasticsearch] is NAS storage.

and this error:

Caused by: java.nio.file.AccessDeniedException: /mnt/elasticsearch/backup/tests-6xN....W_KZQ/data-M.....dat

4

Is /mnt/elasticsearch accessible on that host? Did you configure the path.repo setting?

5

yes, yml file is configured, 2 nodes (..184 and ..185) can write on /mnt/elasticsearch/ but ..183 cant create file on this repo.

6

can you store a file there as the user running elasticsearch in a shell?

7

yes, elasticsearch user can read and write files in a shell, but elasticsearch service can't!!!

8

just to verify: how exactly did you test this?

9

with "sudo -H -u elasticsearch bash -c 'vim test.txt'" command on /mnt/elasticsearch/

10

have you tried to write/read the resource mentioned in the exception?

11

Sorry I don't understand what you mean

12

The exception refers to a file it tries to write at /mnt/elasticsearch/tests-dDepnDzCIA/data-MWHrLmMa_yzbjKthQ.dat - can you try to create that resource via touch on the command line?

13

Thanks for your answer
yes i can do it, when i force node 185 to create file on /mnt/elasticsearch/backup1/ it is done, and when i force node 183 to create file on /mnt/elasticsearch/backup/ it is done too,
but when node 185 is going to create /mnt/elasticsearch/backup/ then node 183 can not read/write files on this dir.
note: primary gid related to elasticsearch on 183 is not equal to node 185, but there is another group on node 183 which is same as primary gid of elasticsearch on 185.
in all nodes UMask of elasticsearch proccess is set to: 0002

14

Thanks for testing. This means, that your system is not set up the same on all nodes. I suppose you need to fix your network storage setup then (or your network file system), as it does not look like an Elasticsearch issue to me. Mapping the GIDs sounds like a good start, but it's hard to tell without knowing the exact setup.

1 Like
15

Thanks for your support

16

if i change primary GID of elasticsearch user, What will be happened?

17

I don't know, as there might be a uid/gid mapping in your network file system that I am not aware off. Also, you may need to change some permissions of your elasticsearch installation (purging the package might make sense in this case). Best would be to test this out in a staging system.

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.