# Glusterfs - Index Replication across sites **URL:** \ **Category:** Elasticsearch\ **Created:** [October 25, 2016, 1:29am UTC](https://discuss.elastic.co/t/glusterfs-index-replication-across-sites/63815 "2016-10-25T01:29:08Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Marcio\_Menezes](https://avatars.discourse-cdn.com/v4/letter/m/e9c0ed/32.png) [@Marcio\_Menezes](https://discuss.elastic.co/u/Marcio_Menezes)\ **Post date:** [October 25, 2016, 1:29am UTC](https://discuss.elastic.co/t/glusterfs-index-replication-across-sites/63815/1 "2016-10-25T01:29:08Z")
In order to have a full active-active cross DC setup, what would be the issues of storing the index in a glusterfs replicated volume. Each DC would read and write to its own local replica and GlusterFS would handle the replication. Does it sound feasible or insane? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 25, 2016, 1:42am UTC](https://discuss.elastic.co/t/glusterfs-index-replication-across-sites/63815/2 "2016-10-25T01:42:33Z")
I don't know the particulars of GlusterFS, but if a write/read needs to traverse a WAN link, ES performance will suffer. ---
**Author:** ![Marcio\_Menezes](https://avatars.discourse-cdn.com/v4/letter/m/e9c0ed/32.png) [@Marcio\_Menezes](https://discuss.elastic.co/u/Marcio_Menezes)\ **Post date:** [October 25, 2016, 1:51am UTC](https://discuss.elastic.co/t/glusterfs-index-replication-across-sites/63815/3 "2016-10-25T01:51:16Z")
Mark, you're right. But in this case, the reads would be local. Only writes would traverse WAN link. In my case, the writes will happen sporadically (once or twice a day), it is basically a CMS that only changes its index when content gets changed and published. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 25, 2016, 1:52am UTC](https://discuss.elastic.co/t/glusterfs-index-replication-across-sites/63815/4 "2016-10-25T01:52:00Z")
> [@Marcio\_Menezes](#): > > Only writes would traverse WAN link. Are the writes synchronous? ---
**Author:** ![Marcio\_Menezes](https://avatars.discourse-cdn.com/v4/letter/m/e9c0ed/32.png) [@Marcio\_Menezes](https://discuss.elastic.co/u/Marcio_Menezes)\ **Post date:** [October 25, 2016, 1:52am UTC](https://discuss.elastic.co/t/glusterfs-index-replication-across-sites/63815/5 "2016-10-25T01:52:20Z")
They would have to be! ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 25, 2016, 1:56am UTC](https://discuss.elastic.co/t/glusterfs-index-replication-across-sites/63815/6 "2016-10-25T01:56:19Z")
So you have one distributed + synchronous write based system, sitting on top of another. Your writes are going to be very slow and may even cause timeouts. You'd need to test. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 25, 2016, 2:02am UTC](https://discuss.elastic.co/t/glusterfs-index-replication-across-sites/63815/7 "2016-10-25T02:02:35Z")
What about the cluster state information that can be found in the data directories? I would expect at least that to cause conflicts if accessed by multiple clusters. If you are indexing/updating rarely, why not just have 2 separate clusters and replicate indices using snapshot and restore? ---
**Author:** ![Marcio\_Menezes](https://avatars.discourse-cdn.com/v4/letter/m/e9c0ed/32.png) [@Marcio\_Menezes](https://discuss.elastic.co/u/Marcio_Menezes)\ **Post date:** [October 25, 2016, 2:11am UTC](https://discuss.elastic.co/t/glusterfs-index-replication-across-sites/63815/8 "2016-10-25T02:11:22Z")
> [@Christian\_Dahlqvist](#): > > If you are indexing/updating rarely, why not just have 2 separate clusters and replicate indices using snapshot and restore? It must be an active-active scenario, meaning that when a change happens, it must be applied on both sites at the same time. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 25, 2016, 2:15am UTC](https://discuss.elastic.co/t/glusterfs-index-replication-across-sites/63815/9 "2016-10-25T02:15:33Z")
> [@Marcio\_Menezes](#): > > at the same time Given the nature of distributed systems, that's _technically_ not possible. So how close to same time do you need to be, can you have N seconds difference? Why can you not have two separate clusters, with your update process writing to both at the same time? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 25, 2016, 2:15am UTC](https://discuss.elastic.co/t/glusterfs-index-replication-across-sites/63815/10 "2016-10-25T02:15:45Z")
In that case the recommended architecture is to distribute the updates separately to the two clusters, e.g. by using a message queue. ---
**Author:** ![Marcio\_Menezes](https://avatars.discourse-cdn.com/v4/letter/m/e9c0ed/32.png) [@Marcio\_Menezes](https://discuss.elastic.co/u/Marcio_Menezes)\ **Post date:** [October 25, 2016, 2:25am UTC](https://discuss.elastic.co/t/glusterfs-index-replication-across-sites/63815/11 "2016-10-25T02:25:22Z")
> [@Christian\_Dahlqvist](#): > > In that case the recommended architecture is to distribute the updates separately to the two clusters, e.g. by using a message queue. So something like Kafka described over here: [Clustering Across Multiple Data Centers | Elastic Blog](https://www.elastic.co/blog/clustering_across_multiple_data_centers) Sounds promising, I just would have to consider failover for this message broker as well. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:10pm UTC](https://discuss.elastic.co/t/glusterfs-index-replication-across-sites/63815/12 "2017-07-05T22:10:08Z")