Thank you for providing me some pointers and sorry for the late reply. We were able to set up a cluster successfully last week.
Having more than one option is a good thing. Let me try to describe a scenario, hopefully that will help you a bit more to answer my question.
We want to create two cluster, one primary and other backup in case the primary goes down for some reason. This means:
- The data needs to be in sync between these two clusters
- When the primary goes down, we switch to secondary cluster. During this time, we will recover the primary. But whatever delta (index updates/new index) is generated during this time on secondary cluster, needs to be restored back to primary as well. In short, both the cluster need to be in sync.
Please note that we will have a separate persistent storage, the scenario we are working on is HA.
As far as what I understand, tribe nodes are not created for this scenario and there is no native replication (master - master / slave like in SQL) support in elasticsearch to achieve the same. We can probably pump in data from Kakfa to primary and secondary clusters, but there is still some work required to handle replication, recovery and validation if we were to create something ourselves to keep two clusters in sync and that seems non-trivial. I wanted to check what are our options.
I hope this gives a good idea about the use case. Let me know if you have more questions.