ES data replication to two ES clusters and restore

Elastic Stack Elasticsearch
1

Hi ES experts,
I have a question as below, please help me take a look, thanks a lot.
we plan to build our disaster recovery(DR) env. we want to save data to our production env and DR
ES clusters at the same time. they are not the same region, So when we do failover to DR, the data is the latest.
I searched below articles and plan to do like these articles mentioned.
Sending logstash data to multiple elastic instances/clusters,
https://stackoverflow.com/questions/41452320/writing-to-multiple-elasticsearch-clusters-from-logstash

My question is: if production env has issue, I failover to DR env, new data will be generated. when production env is restored, i will replication the new data from DR to production, how can i sync the new data to production env from DR? thanks.

2

A common approach to this is to have a queue in place and use two different Logstash pipelines to feed the two clusters off this in parallel. If one cluster has an issue the queue will buffer data until it again becomes available. It will the catch up, but you naturally need to make sure the buffer has enough capacity to handle an outage.

3

Thanks for your reply. I see this option:
Real-time Availability Across Geographic Regions
Here you would have your application code write to a replicated queuing system (e.g. Kafka, Redis, RabbitMQ) and have a process (e.g. Logstash) in each DC reading from the relevant queue and indexing documents into the local Elasticsearch cluster.
This way if network connectivity is lost between the DCs, when it is restored, the indexing will continue where it left off.

as above explained, does this approach look like this? store production data to kafka, the kafka as input, then start each logstash in production env and DR env, echo ES cluster as the output. And another question, do i need start a kafka process on DR env as well, when production env error happened, failover to DR env, application write data to DR kafka, then save data to elasticsearch via logstash. thanks.

4

Hi Christian,
It seems this common approach is focus on to handle if one es cluster down. normally the necessary data is sync to production and hot backup env at the same time via logstash, if my whole production env down, i need to failover to hot backup env, and application run on hot backup, new data will be generated, then after the production recovery, i need to fail back to production env, so how can i sync new data to production from hot backup env? could you give me some suggestions? thanks.

5

Are you just inserting new data or also updating existing documents? Can you describe your use case? How large is your data set?

6

I think both insert new data and update exists documents. my use case as below:
I plan to build a hot backup environment. include all components(all services(java, php, nginx), mysql, es cluster and so on) like production environment , all es data insert to both production and hot backup env. in a word, i hope production and hot backup env have the same es data. when disaster (like all datacenter powered off or earthquake)happened in production env, i can easy failover to hot backup env to provide service. And es data almost is the same as production. when production recovery, i need to sync new data(which generated when failover to hot backup) to production env from hot backup env. i'm not sure whether you understand my question, thanks for your patience.

7

I do not think there is any easy way to create an active-active environment that can handle inserts as well as updates without queueing up write operations for both clusters like I described. Maybe someone else have some suggestions.

8

Thanks. It's not easy to achieve active-active environment. so I opened this topic to seek some suggestions from es experts. Anyway first I will try to use kafka as queuing system and logstash output to both es clusters.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.