Replicate Data Asynchron

[JonasPlaum]

Hello,

i have a question, about how i can design my data analysis. At the moment we have services that write a bunch of Events to mariaDB. MariaDB is replicating the data between 2 or more Database Nodes. There are multiple regions that generate Events. Every Region has one or more MariaDB Node, where the local services write to. Because of legal requirements it has to be possible to write Data to any DB Node, even if the Region is seperated from the other regions. Because of this its not possible to use syncronous replication between the database nodes. MariaDB can replicate the data after the connection is restored. After the replication data gets read from MariaDB from one of our services and written to two elasticsearch Node per region, so that we can visualize them with kibana.

We now want to replace or better remove the mariaDB step. Here is the point were I need some help.
Is it possible to replicate data between elasticsearch Nodes asynchron, so that data could be written in case of Network separation and later replicated? If so we can use elastic only for our usecase.

If not, I need an other way of replicating the data. What is a good way to do this, that works well with elasticsearch. Im thinking about using a NoSQL Database to store the Data and handle replication, but i dont want to write an service that moves data from there to elasticsearch. I think there should be tools that can handle this for me.

Or are there other ideas how i can handle this?

Thanks for your help and any ideas.

[Christian_Dahlqvist]

If you need to be able to handle the regions becoming disconnected and still support indexing in both regions you will not be able to use a single Elasticsearch cluster. If you instead have two separate clusters a common way to replicate data across the clusters is to add Kafka message queues and have separate processes reading from this and indexing into the two local clusters. If there is a network partition it will buffer data which the writer can then access in order to catch up when connectivity is restored.

Another option is cross-cluster replication which may be an option depending on your data, but be aware that this is a feature that requires a commercial license.