Multiple Clusters: Replicating data to Another ES Cluster

[animageofmine (Ankit Parikh)]

Hello,

We are planning to use Elasticsearch on production and were wondering what is the suggested practice for HA (High Availability) or BCP (Business Continuity Plan).

1. What is the recommended practice for handling downtime for elasticsearch cluster (power loss, maintenance, upgrade, patch)?
2. How long does it take to upgrade (e.g. from version 2.4 to 5 or from 5.0.1 to 5.0.2)?
3. Does minor version upgrades also require cluster restart?
4. Is there a way to replicate data from one cluster to another if we were to create a backup cluster?

FYI, our cluster is hosted (not on AWS). Thank you for your help.

[eperry (Ed)]

I could write a book on these questions. There is no one easy answer to them all except narrow your questions to something more specific, or sit down with a sales engineer and talk over all these things.

But not to leave you empty handed.

1. there are many architectures you can implement for redundancy, rack aware, data center awayness, Tribe nodes. etc
   a. Patching of ELK is usually done as a rolling upgrade - you would have to read the release notes on any upgrade. And it is well documented how to do rolling upgrades
   https://www.elastic.co/guide/en/elasticsearch/reference/current/rolling-upgrades.html

2. See 1.a Elastic is in explosive growth right now and major version do have breaking changes. and they do warn that minor version could also have breaking changes. ( read your release notes )

3. yes see 1.a

4. Yes depends on the type of backup you want, everything from replication away DC/Rack, to multi-cluster/tribes to snapshots can be made, heck even updating 2 different clusters. Might give you some ideas https://www.elastic.co/blog/clustering_across_multiple_data_centers
   a. Depending on the size and indexing behavior you can do things with Kafka and different logstash to publish to two completely independent clusters.

There are really too many options to really accurately describe all the options. I think you would be best talking to a Sales Engineer even if you don't purchase the licenses.

I would be glad to keep talking if you need clarifications on the above. Or if you have a requirements list vs, asking what is possible maybe I or someone else can enlighten you on options related to your specific needs.

[animageofmine (Ankit Parikh)]

Thank you for providing me some pointers and sorry for the late reply. We were able to set up a cluster successfully last week.

Having more than one option is a good thing. Let me try to describe a scenario, hopefully that will help you a bit more to answer my question.

We want to create two cluster, one primary and other backup in case the primary goes down for some reason. This means:

1. The data needs to be in sync between these two clusters
2. When the primary goes down, we switch to secondary cluster. During this time, we will recover the primary. But whatever delta (index updates/new index) is generated during this time on secondary cluster, needs to be restored back to primary as well. In short, both the cluster need to be in sync.

Please note that we will have a separate persistent storage, the scenario we are working on is HA.

As far as what I understand, tribe nodes are not created for this scenario and there is no native replication (master - master / slave like in SQL) support in elasticsearch to achieve the same. We can probably pump in data from Kakfa to primary and secondary clusters, but there is still some work required to handle replication, recovery and validation if we were to create something ourselves to keep two clusters in sync and that seems non-trivial. I wanted to check what are our options.

I hope this gives a good idea about the use case. Let me know if you have more questions.