# Cross DC Cluster and Data Replication **URL:** \ **Category:** Elasticsearch\ **Created:** [December 29, 2017, 6:05am UTC](https://discuss.elastic.co/t/cross-dc-cluster-and-data-replication/113532 "2017-12-29T06:05:11Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![senthil.nathanm](https://avatars.discourse-cdn.com/v4/letter/s/f475e1/32.png) [@senthil.nathanm](https://discuss.elastic.co/u/senthil.nathanm)\ **Post date:** [December 29, 2017, 6:05am UTC](https://discuss.elastic.co/t/cross-dc-cluster-and-data-replication/113532/1 "2017-12-29T06:05:11Z")
Hi All We are in the process of establishing Elastic Search on enterprise scale which will span servers spread across two primary data centers located in US and Europe. We are using 5.5.x version and our goal is to have a single view of enterprise data spread across both DCs. I am trying to ascertain the right approach to go about this. Will a single cluster spanning DCs be feasible ? If not, what kind of replication scheme be used for replicating data between the clusters located in each DC ? Do you recommend using Kafka as a queuing mechanism or use logstash's own persistent queues instead ? Regards Senthil Nathan M ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 29, 2017, 6:09am UTC](https://discuss.elastic.co/t/cross-dc-cluster-and-data-replication/113532/2 "2017-12-29T06:09:38Z")
> [@senthil.nathanm](#): > > Will a single cluster spanning DCs be feasible ? No. > [@senthil.nathanm](#): > > If not, what kind of replication scheme be used for replicating data between the clusters located in each DC ? Depends what you really need here. > [@senthil.nathanm](#): > > Do you recommend using Kafka as a queuing mechanism or use logstash's own persistent queues instead ? Again, depends on your needs. What is the replication actually hoping to achieve? We are working on functionality to allow native cross cluster replication. It should be available in the near future. ---
**Author:** ![senthil.nathanm](https://avatars.discourse-cdn.com/v4/letter/s/f475e1/32.png) [@senthil.nathanm](https://discuss.elastic.co/u/senthil.nathanm)\ **Post date:** [December 29, 2017, 9:21am UTC](https://discuss.elastic.co/t/cross-dc-cluster-and-data-replication/113532/3 "2017-12-29T09:21:54Z")
Thanks for the prompt response Mark. Requirements: Our primary objective here is to have an elastic search environment that can aggregate various log files from all servers in both data centers (US and EU). We are looking for a unified view of all the collected log messages (via kibana or similar interface). The search and indexing latency must be minimal. I looked at [this](https://www.elastic.co/blog/scaling_elasticsearch_across_data_centers_with_kafka) page and felt "Independent Elasticsearch and Kafka Clusters" option suited our requirements better. Questions: For our requirements, does having a dedicated Elastic Search cluster and kafka instance in each data center be appropriate ? If so, can I use a simple logstash service to replicate data from remote DC ? That page was over 2 years old and with elastic search having added more features now, I wanted a fresh perspective on the architecture discussed there. With the features available in 5.5.x, is the architecture discussed in the page still viable ? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 29, 2017, 9:23am UTC](https://discuss.elastic.co/t/cross-dc-cluster-and-data-replication/113532/4 "2017-12-29T09:23:05Z")
Why not use 6.X and cross cluster search? ---
**Author:** ![senthil.nathanm](https://avatars.discourse-cdn.com/v4/letter/s/f475e1/32.png) [@senthil.nathanm](https://discuss.elastic.co/u/senthil.nathanm)\ **Post date:** [December 29, 2017, 9:27am UTC](https://discuss.elastic.co/t/cross-dc-cluster-and-data-replication/113532/5 "2017-12-29T09:27:07Z")
Thanks Mark. I'll look into the feasibility of us upgrading to 6.x. Meanwhile, if there is something we can accomplish with 5.5.x, kindly let me know. Appreciate your prompt response. Regards Senthil Nathan M ---
**Author:** ![senthil.nathanm](https://avatars.discourse-cdn.com/v4/letter/s/f475e1/32.png) [@senthil.nathanm](https://discuss.elastic.co/u/senthil.nathanm)\ **Post date:** [January 5, 2018, 2:19pm UTC](https://discuss.elastic.co/t/cross-dc-cluster-and-data-replication/113532/6 "2018-01-05T14:19:48Z")
Hi Mark Our planned setup works like this. America DC: (1) Client Nodes (2) Logstash Node (the logs generated by all clients in this DC is processed by this logstash node) Europe DC: (1) Client Nodes (2) Logstash Nodes (the logs generated by all clients in this DC is processed by this logstash node) (3) Elastic Search Server (4) Elastic Search Database (5) Kibana The data collected by logstash in America DC will be transferred to the Elastic Search Server in Europe DC. Also, the elastic search volume is expected to expand rapidly this year, Do you foresee any problems with this architecture that would affect elastic search performance or is it sustainable ? Regards Senthil Nathan M ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 5, 2018, 8:02pm UTC](https://discuss.elastic.co/t/cross-dc-cluster-and-data-replication/113532/7 "2018-01-05T20:02:35Z")
That looks ok. ---
**Author:** ![senthil.nathanm](https://avatars.discourse-cdn.com/v4/letter/s/f475e1/32.png) [@senthil.nathanm](https://discuss.elastic.co/u/senthil.nathanm)\ **Post date:** [January 8, 2018, 6:37am UTC](https://discuss.elastic.co/t/cross-dc-cluster-and-data-replication/113532/8 "2018-01-08T06:37:51Z")
Thank you Mark for the prompt response. I am happy to close this thread as I got the answer I was looking for. Regards Senthil Nathan M ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 5, 2018, 6:38am UTC](https://discuss.elastic.co/t/cross-dc-cluster-and-data-replication/113532/9 "2018-02-05T06:38:06Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.