i would like to do an offline backup of nodes. Say, i stop elasticsearch on all nodes, rsync /data/elasticsearch/nodes/ to a remote NFS server. would i get the same data(indices, visualization,etc) if i rsync from NFS server to a different ES host?
It's not recommended. From the docs:
WARNING: You cannot back up an Elasticsearch cluster by simply taking a copy of the data directories of all of its nodes [...] The only reliable way to back up a cluster is by using the snapshot and restore functionality.
You can get into a real mess with other backup schemes. For instance, if you do as you describe then your two clusters would have the same cluster UUID; if they ever accidentally touched each other then you would likely lose data in the resulting confusion. What's wrong with taking a snapshot to a NFS server and then restoring it into another cluster elsewhere? This is the supported method for backups and is designed to avoid such catastrophes. It also works without shutting the whole cluster down.
Thanks @DavidTurner. Here is the actual scenario.
my /data/elasticsearch/nodes/ is in a non-encrypted logical volume. I want to put them on an encypted volume. So I'm thinking of ,
- stop elasticsearch
- rsync nodes/ out to NFS
- delete /data
- re-created /data with encryption
- rsync back from NFS
- start elasticsearch
I see. Yes, moving the entire contents of the data path elsewhere (and destroying the original) is ok if the node is shut down. It's copying it (i.e. keeping the original) where you get problems. Make sure you preserve everything, including file ownerships and permissions and so on, of course.
Thanks much @DavidTurner