ELK CCR Setup

[mostafaelsayed]

Hello,

I want to test the CCR feature, and I understand these two limitations exist:

1. A follower can only follow one leader.
2. No way to directly write events from the client side to the follower index.

with these two limitations, whether the setup is Active-Standby or Active-Active, there must be some events dropped in case of outage because the leader in the active region is down, and the follower in the passive region cannot be written to.
(I am using both time-based indices and non-time-based indices and document updates are happening).

Is there any way to prevent this from happening instead of manually switching the active and standby indices? The availability aspect is important for our use case so we want to make sure we don't lose any events as much as we can in case of outage in a region.

Thanks

[stephenb (Stephen Brown)]

Hi @mostafaelsayed

Take a look at this it is a little dated but concepts still apply.

Basically, you have a leader/follower on both sides...

Each Side write to its leader

Each side Read from a Data view that reads both leader and follower...

These two images shows the concept

Write Alias always writes to leader

Read Alias always reads from both

So Imagine dc1 goes down you just switch the inbound traffic to the write alias in DC2...

This can work in the Active Standby... When Active goes down then you write to the write alias in the Standby... and read from both.

Obviously, there are details and different implementations and you could use a load balance with a health check etc... but that is the basic concept.

BTW this is not without some additional work, Elastic provides building blocks for these types of operations but it is not native out of the box, I am hoping we make this simpler for our users in the future.

[mostafaelsayed]

Hi @stephenb

Thanks. But the problem is that the application is doing document updates so this setup still won't work for this case.

[stephenb (Stephen Brown)]

Hi @mostafaelsayed

Yes as the article says this does not support updates, apologies, your post said write not update I made a poor assumption.

Bidirectional updates are not supported.

Instant failover of write/update to a follower index is not supported.

Neither of these is supported as far as I understand at this time, it takes time to "unfollow" a leader and then it can not be turned back into a follower again.

The users/customer that I work with that have such stringent requirements use a queuing technology like Kafka to support these types of requirements

[mostafaelsayed]

Thanks @stephenb I will check out the Kafka option

[mostafaelsayed]

Hello @stephenb

I want to confirm something please.

the following image from the docs suggests that a follower can follow more than one leader which I think this is a limitation and can't happen, right?

Is this true that one follower can follow multiple leaders, or the image is assuming that no updates happen, and this one follower is in fact multiple followers, one for each leader?

Thanks

[Christian_Dahlqvist]

I believe the diagram assumes you have one index per region and that the follower is replicating these 3 indices (different names) into the central reporting cluster.

[mostafaelsayed]

Thank you @Christian_Dahlqvist I believe that too, I just wanted to confirm.