Fs snapshot permission issues inside a kubernetes cluster

Elastic Stack Elasticsearch
1

Hi Team,

I'm trying to take snapshot of elasticsearch which is deployed inside a kubernetes cluster.

I'm trying to take backup on file system whereas Im getting below error.
I understand that the default user will be elasticsearch, so while mounting the volume
Error:
{ "type": "repository_verification_exception", "reason": "[my_backup] [[Vy3xhG9OQ5SaHo5oIEYK0w, 'RemoteTransportException[[elasticsearch-master-1][10.233.93.69:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_backup] a file written by master to the store [/usr/share/elasticsearch/backups] cannot be accessed on the node [{elasticsearch-master-1}{Vy3xhG9OQ5SaHo5oIEYK0w}{sflWWtUaTw-qeecII-tdTg}{10.233.93.69}{10.233.93.69:9300}]. This might indicate that the store [/usr/share/elasticsearch/backups] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node];'], [K89SLsu7RMS8fIey2P0ihw, 'RemoteTransportException[[elasticsearch-master-0][10.233.102.120:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_backup] a file written by master to the store [/usr/share/elasticsearch/backups] cannot be accessed on the node [{elasticsearch-master-0}{K89SLsu7RMS8fIey2P0ihw}{WP14RXVoQemhwaFFUYKRcg}{10.233.102.120}{10.233.102.120:9300}]. This might indicate that the store [/usr/share/elasticsearch/backups] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node];']]" }

image
image955×473 75.8 KB

image
image1779×742 120 KB

Also including the volumeMount security context:
"securityContext": { "privileged": true, "runAsUser": 1000 }
Any leads would be more helpful.

Thanks in Advance!

2

Please don't post pictures of text, they are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them :slight_smile:

A volume/path mounted for snapshots must be available - read+write - on all nodes in the cluster. It cannot just be a local path on a single node.

3

Hi Mark,

Sorry for the inconvenience. I have deployed Elastic search as Statefulset, I wonder how the volume is not shared.

Thanks
Rajashree

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.