Snapshot/restore metadata file access

Elastic Stack Elasticsearch
1

Hello -

I have two clusters set up with dedicated master nodes and data nodes. With cluster A running in one data center and cluster B running in another data center. There is NAS storage mounted to the master and data nodes that will be used as the backing store for the snapshot/restore process.

From what I understand, during the snapshot/restore process, the master nodes create/read/write the metadata- and snapshot- files and the data nodes create/read/write the index files.

My question is, do the data nodes ever read/write anything to/from the metadata- and snapshot- files either during the snapshot or restore process? Or, are those metadata- files only read/written to by the master nodes?

I ask, because I have a need to provide encryption of the index files at rest due to the nature of the data stored in the cluster. I am ultimately wondering if I can apply the encryption only to the files written by the data nodes or whether I will need to use the same encryption techniques on the master nodes as well so that the data nodes can read the metadata- and snapshot- files.

Regards,
Chris

2

The elected master node is the only node that can modify the metadata, what we usually call cluster state. Any node can trigger a cluster state update (e.g. index creation), but that request will always be rerouted to the elected master node, which executes cluster state updates sequentially. After each update the new cluster state will be pushed to all of the nodes in the cluster. To wrap it up, each node holds the cluster state, but only the master modifies it.

With that I am not sure this answers your question 100%, maybe it will help expanding it so we can better answer it?

Cheers

3

javanna thanks for the response.
Wasn't quite the answer I was looking for. In particular, when I perform a snapshot on my cluster I get the following folders/files created on my shared NAS:

user1@xxxxxxx:/elasticsearch/backup/app1/indices/index1> ls -ltrtotal 20-rw-rw-rw- 1 user1 user1 7709 Dec 17 17:43 snapshot-audit-12_17_2015t17_43_35_165013129drwxrwsrwx 2 user1 user1 4096 Dec 17 17:44 2drwxrwsrwx 2 user1 user1 4096 Dec 17 21:48 0drwxrwsrwx 2 user1 user1 4096 Dec 17 22:31 1

The snapshot-* file contains the following information:
{"index1":{"version":12,"state":"open","settings":{"index.creation_date":"1447974957071","index.uuid":"CsQKGpzpRkCxG4ObyhKaJQ","index.version.created":"1070299","index.number_of_replicas":"1","index.number_of_shards":"3"},"mappings":[{"RedactionType":{"properties":{"action":{"type":"string","fields":{"raw":{"type":"string","index":"not_analyzed","doc_values":true}}},"auditId":{"type":"string","fields":{"raw":{"type":"string","index":"not_analyzed","doc_values":true}}},"customerStorageId":{"type":"string","fields":{"raw":{"type":"string","index":"not_analyzed","doc_values":true}}},"matterId":{"type":"string","fields":{"raw":{"type":"string","index":"not_analyzed","doc_values":true}}},"modifiedByDataRoomContactId":{"type":"string","fields":{"raw":{"type":"string","index":"not_analyzed","doc_values":true}}},"modifiedByFirstName":{"type":"string","fields":{"raw":{"type":"string","index":"not_analyzed","doc_values":true}}},"modifiedByFullName":{"type":"string","fields":{"raw":{"type":"string","index":"not_analyzed","doc_values":true}}},"modifiedByLastName":{"type":"string","fields":{"raw":{"type":"string","index":"not_analyzed","doc_values":true}}},"modifiedByUserSessionId":{"type":"string","fields":{

Which node types (master / data) are allowed to write to the "snapshot-audit-12_17_2015t17_43_35_165013129" file?

4

Any and all nodes that have data pertaining to the index write to the files.

5

Thank you very much, this helps.