Glusterfs - Index Replication across sites

(Márcio Menezes) #1

In order to have a full active-active cross DC setup, what would be the issues of storing the index in a glusterfs replicated volume.
Each DC would read and write to its own local replica and GlusterFS would handle the replication.
Does it sound feasible or insane?

(Mark Walkom) #2

I don't know the particulars of GlusterFS, but if a write/read needs to traverse a WAN link, ES performance will suffer.

(Márcio Menezes) #3

Mark, you're right.

But in this case, the reads would be local. Only writes would traverse WAN link.
In my case, the writes will happen sporadically (once or twice a day), it is basically a CMS that only changes its index when content gets changed and published.

(Mark Walkom) #4

Are the writes synchronous?

(Márcio Menezes) #5

They would have to be!

(Mark Walkom) #6

So you have one distributed + synchronous write based system, sitting on top of another.
Your writes are going to be very slow and may even cause timeouts.

You'd need to test.

(Christian Dahlqvist) #7

What about the cluster state information that can be found in the data directories? I would expect at least that to cause conflicts if accessed by multiple clusters. If you are indexing/updating rarely, why not just have 2 separate clusters and replicate indices using snapshot and restore?

(Márcio Menezes) #8

It must be an active-active scenario, meaning that when a change happens, it must be applied on both sites at the same time.

(Mark Walkom) #9

Given the nature of distributed systems, that's technically not possible.
So how close to same time do you need to be, can you have N seconds difference?

Why can you not have two separate clusters, with your update process writing to both at the same time?

(Christian Dahlqvist) #10

In that case the recommended architecture is to distribute the updates separately to the two clusters, e.g. by using a message queue.

(Márcio Menezes) #11

So something like Kafka described over here:

Sounds promising, I just would have to consider failover for this message broker as well.

(system) #12