SAN index replication from PROD to DR(disaster recovery) server

Elastic Stack Elasticsearch
1

Hi,

I am facing few issues in replicating my SAN index to DR server from my PROD index. For NAS index its perfectly fine we are able to replicate but for SAN we are not. We have used the OS rsync method to copy index from PROD to DR but it fails sometimes.

Is their any way to do SAN replication. Any guidance and help is welcome

P.S. PROD and DR are different servers

Thanks
Satya

2

Perhaps periodic snapshots from ES may be a better approach -
https://www.elastic.co/guide/en/elasticsearch/guide/current/backing-up-your-cluster.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html

Using rsync I think would generally be problematic unless you can flush and create an filesystem-level snapshot (ala ZFS) and rsync that over to the DR server. Otherwise, you're attempting to sync files that are in a constant state of flux.

3

Hi Bruce,

Snapshots is fine but we have index in SAN(local drive) and not NAS(network drive). So how will we replicate it from PROD to DR as SAN is not over network. Do you mean to say create local snapshots in PROD and then manually push them to DR everyday ?

Thanks
Satya

4

Satya,

That was my first thought, yes. Have a local snapshot directory in PROD that you rsync to DR and use that for restoring the index into DR. It may not be the most efficient method but I'm fairly certain it'll be consistent and stable. If you need more real-time DR than what you can get with snapshot/restore you may have to introduce middleware to synchronize the ES instances. See https://www.elastic.co/blog/scaling_elasticsearch_across_data_centers_with_kafka for an example of what that might look like.

Bruce