So, my work requires georedundancy for every solution. It's not good enough that things like elasticsearch are not designed for that, and in fact, spanning-datacenter-clusters actually open you to more potential issues. I completely understand and agree with the reasoning behind that. Looks like some people do it anyway: http://gibrown.com/2014/01/09/scaling-elasticsearch-part-1-overview/
However, we still need to have somewhat of a solution. Active/active is preferred, but hot standby is a minimum requirement.
I see a couple ways to approach this: single cluster, and multi-cluster. Single cluster meaning the data nodes in the secondary data center are part of the same cluster. Multi-cluster means there is some synchronization process to make sure the standby cluster has all of the data in the primary cluster. We do have a dedicated link with a huge pipe, but it is safe to assume that the network will fail at some point. There are things like couchbase which can automatically synchronize cluster data (http://blog.couchbase.com/announcing-release-couchbase-plug-elasticsearch), but that seems like quite a bit of technical debt to take on.
Has anyone had experience with running a georedundant solution? Or any avenues for research?
Right now I am leaning toward snapshot/restore, because we'll need to be taking snapshots anyway. We just need a process to automatically restore the snapshots on the primary to the secondary cluster(s). Am I overlooking anything?