# Cross data center replication **URL:** \ **Category:** Elasticsearch\ **Created:** [April 23, 2013, 3:22pm UTC](https://discuss.elastic.co/t/cross-data-center-replication/11670 "2013-04-23T15:22:38Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Saikat\_Kanjilal](https://avatars.discourse-cdn.com/v4/letter/s/c0e974/32.png) [@Saikat\_Kanjilal](https://discuss.elastic.co/u/Saikat_Kanjilal)\ **Post date:** [April 23, 2013, 3:22pm UTC](https://discuss.elastic.co/t/cross-data-center-replication/11670/1 "2013-04-23T15:22:38Z")
Hello Folks, I digged through the documentation and found a few wiki posts in places but nothing that seems to answer this question directly, as of the latest release of does ES out of the box currently support cross data center replication, I've seen a post or two regarding use cases where folks are running ES on top of a key value store that supports this like Couchbase but nothing to indicate that ES itself has support for this. Some insight or links to docs regarding this would be very helpful. Thanks in advance. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Daniel\_Maher\_3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniel_maher_3/32/2014_2.png) [@Daniel\_Maher\_3](https://discuss.elastic.co/u/Daniel_Maher_3)\ **Post date:** [April 23, 2013, 3:44pm UTC](https://discuss.elastic.co/t/cross-data-center-replication/11670/2 "2013-04-23T15:44:44Z")
On 2013-04-23 5:22 PM, Saikat Kanjilal wrote: > Hello Folks, > I digged through the documentation and found a few wiki posts in places > but nothing that seems to answer this question directly, as of the > latest release of does ES out of the box currently support cross data > center replication, I've seen a post or two regarding use cases where > folks are running ES on top of a key value store that supports this like > Couchbase but nothing to indicate that ES itself has support for this. > Some insight or links to docs regarding this would be very helpful. Hello, I'd wager that the question you're really asking about is how to control where shards are placed; if you can make deterministic statements about where shards are, then you can create your own "rack-aware" or "data centre-aware" scenarios. ES has supported this "out of the box" for well over a year now (possibly longer). You'll want to investigate "zones" and "routing allocation", which are the key elements of shard placement. There is an excellent blog post which describes exactly how to set things up here : > **[ElasticSearch Shard Placement Control - Sematext](https://sematext.com/blog/elasticsearch-shard-placement-control/)** > > In this post you will learn how to control ElasticSearch shard placement.  If you are coming to our ElasticSearch talk at BerlinBuzzwords, we’ll be talking about this and more. If you’ve ever used ElasticSearch you probably know that you can set... Enjoy ! -- dan (phrawzty). mozilla webops; european outpost. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Norberto\_Meijome](https://avatars.discourse-cdn.com/v4/letter/n/ed655f/32.png) [@Norberto\_Meijome](https://discuss.elastic.co/u/Norberto_Meijome)\ **Post date:** [April 24, 2013, 10:37am UTC](https://discuss.elastic.co/t/cross-data-center-replication/11670/3 "2013-04-24T10:37:42Z")
Along the same line, once you have your zones and shards allocated across different DCs, is it possible to have queries originating on DC #1 to only stay in DC #1 ? ie, how can we control the ES nodes from distributing the queries across all nodes. Alternatively, is there a way to tell the ES cluster about 'shard distance' (so that queries are optimised where shard distance is minimised ) ? Thanks!! Beto On Wed, Apr 24, 2013 at 1:44 AM, Daniel Maher [dmaher@mozilla.com](mailto:dmaher@mozilla.com) wrote: > On 2013-04-23 5:22 PM, Saikat Kanjilal wrote: > > > Hello Folks, > > I digged through the documentation and found a few wiki posts in places > > but nothing that seems to answer this question directly, as of the > > latest release of does ES out of the box currently support cross data > > center replication, I've seen a post or two regarding use cases where > > folks are running ES on top of a key value store that supports this like > > Couchbase but nothing to indicate that ES itself has support for this. > > Some insight or links to docs regarding this would be very helpful. > > Hello, > > I'd wager that the question you're really asking about is how to control > where shards are placed; if you can make deterministic statements about > where shards are, then you can create your own "rack-aware" or "data > centre-aware" scenarios. ES has supported this "out of the box" for well > over a year now (possibly longer). > > You'll want to investigate "zones" and "routing allocation", which are the > key elements of shard placement. There is an excellent blog post which > describes exactly how to set things up here : > [http://blog.sematext.com/2012/\*\*05/29/elasticsearch-shard-](http://blog.sematext.com/2012/**05/29/elasticsearch-shard-)\*\* > placement-control/[http://blog.sematext.com/2012/05/29/elasticsearch-shard-placement-control/](http://blog.sematext.com/2012/05/29/elasticsearch-shard-placement-control/) > > Enjoy ! > > -- > dan (phrawzty). > mozilla webops; european outpost. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com) > . > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > . -- Norberto 'Beto' Meijome -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [April 24, 2013, 11:26am UTC](https://discuss.elastic.co/t/cross-data-center-replication/11670/4 "2013-04-24T11:26:15Z")
Hello Beto, I didn't use this feature (yet), but you have some options you can specify at query time for shard preference: > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene On Wed, Apr 24, 2013 at 1:37 PM, Norberto Meijome [numard@gmail.com](mailto:numard@gmail.com) wrote: > Along the same line, once you have your zones and shards allocated across > different DCs, is it possible to have queries originating on DC #1 to only > stay in DC #1 ? ie, how can we control the ES nodes from distributing the > queries across all nodes. Alternatively, is there a way to tell the ES > cluster about 'shard distance' (so that queries are optimised where shard > distance is minimised ) ? > > Thanks!! > Beto > > On Wed, Apr 24, 2013 at 1:44 AM, Daniel Maher [dmaher@mozilla.com](mailto:dmaher@mozilla.com) wrote: > > > On 2013-04-23 5:22 PM, Saikat Kanjilal wrote: > > > > > Hello Folks, > > > I digged through the documentation and found a few wiki posts in places > > > but nothing that seems to answer this question directly, as of the > > > latest release of does ES out of the box currently support cross data > > > center replication, I've seen a post or two regarding use cases where > > > folks are running ES on top of a key value store that supports this like > > > Couchbase but nothing to indicate that ES itself has support for this. > > > Some insight or links to docs regarding this would be very helpful. > > > > Hello, > > > > I'd wager that the question you're really asking about is how to control > > where shards are placed; if you can make deterministic statements about > > where shards are, then you can create your own "rack-aware" or "data > > centre-aware" scenarios. ES has supported this "out of the box" for well > > over a year now (possibly longer). > > > > You'll want to investigate "zones" and "routing allocation", which are > > the key elements of shard placement. There is an excellent blog post which > > describes exactly how to set things up here : > > [http://blog.sematext.com/2012/\*\*05/29/elasticsearch-shard-](http://blog.sematext.com/2012/**05/29/elasticsearch-shard-)\*\* > > placement-control/[http://blog.sematext.com/2012/05/29/elasticsearch-shard-placement-control/](http://blog.sematext.com/2012/05/29/elasticsearch-shard-placement-control/) > > > > Enjoy ! > > > > -- > > dan (phrawzty). > > mozilla webops; european outpost. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com) > > . > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > . > > -- > Norberto 'Beto' Meijome > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Norberto\_Meijome](https://avatars.discourse-cdn.com/v4/letter/n/ed655f/32.png) [@Norberto\_Meijome](https://discuss.elastic.co/u/Norberto_Meijome)\ **Post date:** [April 28, 2013, 11:49pm UTC](https://discuss.elastic.co/t/cross-data-center-replication/11670/5 "2013-04-28T23:49:13Z")
thanks guys, yeah, i had noticed those options - it seems I'll need an external service to synchronise the state of the ES cluster to the app ( ZK ) ... given there is only an option to specify node by node\_id, rather than by a property of the node ( where it is located,for example). does sound like an useful thing to have, imo - in case it isn't obvious, I'm running ES on AWS. cheers, Beto On Wed, Apr 24, 2013 at 9:26 PM, Radu Gheorghe [radu.gheorghe@sematext.com](mailto:radu.gheorghe@sematext.com)wrote: > Hello Beto, > > I didn't use this feature (yet), but you have some options you can specify > at query time for shard preference: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/preference/) > > ## Best regards, Radu > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > On Wed, Apr 24, 2013 at 1:37 PM, Norberto Meijome [numard@gmail.com](mailto:numard@gmail.com)wrote: > > > Along the same line, once you have your zones and shards allocated across > > different DCs, is it possible to have queries originating on DC #1 to only > > stay in DC #1 ? ie, how can we control the ES nodes from distributing the > > queries across all nodes. Alternatively, is there a way to tell the ES > > cluster about 'shard distance' (so that queries are optimised where shard > > distance is minimised ) ? > > > > Thanks!! > > Beto > > > > On Wed, Apr 24, 2013 at 1:44 AM, Daniel Maher [dmaher@mozilla.com](mailto:dmaher@mozilla.com) wrote: > > > > > On 2013-04-23 5:22 PM, Saikat Kanjilal wrote: > > > > > > > Hello Folks, > > > > I digged through the documentation and found a few wiki posts in places > > > > but nothing that seems to answer this question directly, as of the > > > > latest release of does ES out of the box currently support cross data > > > > center replication, I've seen a post or two regarding use cases where > > > > folks are running ES on top of a key value store that supports this like > > > > Couchbase but nothing to indicate that ES itself has support for this. > > > > Some insight or links to docs regarding this would be very helpful. > > > > > > Hello, > > > > > > I'd wager that the question you're really asking about is how to control > > > where shards are placed; if you can make deterministic statements about > > > where shards are, then you can create your own "rack-aware" or "data > > > centre-aware" scenarios. ES has supported this "out of the box" for well > > > over a year now (possibly longer). > > > > > > You'll want to investigate "zones" and "routing allocation", which are > > > the key elements of shard placement. There is an excellent blog post which > > > describes exactly how to set things up here : > > > [http://blog.sematext.com/2012/\*\*05/29/elasticsearch-shard-](http://blog.sematext.com/2012/**05/29/elasticsearch-shard-)\*\* > > > placement-control/[http://blog.sematext.com/2012/05/29/elasticsearch-shard-placement-control/](http://blog.sematext.com/2012/05/29/elasticsearch-shard-placement-control/) > > > > > > Enjoy ! > > > > > > -- > > > dan (phrawzty). > > > mozilla webops; european outpost. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com) > > > . > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > . > > > > -- > > Norberto 'Beto' Meijome > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Norberto 'Beto' Meijome -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![phill](https://avatars.discourse-cdn.com/v4/letter/p/779978/32.png) [@phill](https://discuss.elastic.co/u/phill)\ **Post date:** [April 30, 2013, 8:55pm UTC](https://discuss.elastic.co/t/cross-data-center-replication/11670/6 "2013-04-30T20:55:02Z")
On 4/23/2013 8:44 AM, Daniel Maher wrote: > On 2013-04-23 5:22 PM, Saikat Kanjilal wrote: > > > Hello Folks, > > [...] does ES out of the box currently support cross data > > center replication, [....] > > Hello, > > I'd wager that the question you're really asking about is how to > control where shards are placed; if you can make deterministic > statements about where shards are, then you can create your own > "rack-aware" or "data centre-aware" scenarios. ES has supported this > "out of the box" for well over a year now (possibly longer). > > You'll want to investigate "zones" and "routing allocation", which are > the key elements of shard placement. There is an excellent blog post > which describes exactly how to set things up here : > [ElasticSearch Shard Placement Control - Sematext](http://blog.sematext.com/2012/05/29/elasticsearch-shard-placement-control/) Is shard allocation really the correct solution if the data centers are globally distributed? If I have a data center in the US intended to server data from the US, but it should also have access to Europe and Asia data, and clusters in both Europe and Asia with similar needs, would I really want to use zones etc. and have one great global cluster with data center aware configurations? Assuming that the US would be happy to deal with old documents from Asia and Europe, when Asia or Europe is off line or just not caught up, it would seem that you would NOT want a "world" cluster, because I can't picture how you'd configure a 3-part world cluster for both index into the right indices, search the right (possible combination of) shards, but also preventing "split brain". In the scenerio, I've described, I would think each data center might better provide availability and eventual consistency (with less concern for the remote data from the other region) by having three clusters and some type of syncing from one index to copies at the other two locations. For example, the US datacenter might have a US, copyOfEurope, and copyOfAsia index. Anyone have any observations about such a world-wide scenerio? Are there any index to index copy utilities? Is there a river or other plugin that might be useful for this three clusters working together scenerio? How about the project [GitHub - karussell/elasticsearch-reindex: Simple re-indexing. To backup, apply index settings changes and more ElasticMagic](https://github.com/karussell/elasticsearch-reindex)? Comments? -Paul -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Norberto\_Meijome](https://avatars.discourse-cdn.com/v4/letter/n/ed655f/32.png) [@Norberto\_Meijome](https://discuss.elastic.co/u/Norberto_Meijome)\ **Post date:** [May 1, 2013, 12:50pm UTC](https://discuss.elastic.co/t/cross-data-center-replication/11670/7 "2013-05-01T12:50:00Z")
+1 on all of the above. es-reindex already in my list of things to investigate (for a number of issues...) cheers, b On Wed, May 1, 2013 at 6:58 AM, Paul Hill [parehill1@gmail.com](mailto:parehill1@gmail.com) wrote: > On 4/23/2013 8:44 AM, Daniel Maher wrote: > > > On 2013-04-23 5:22 PM, Saikat Kanjilal wrote: > > > > > Hello Folks, > > > [...] does ES out of the box currently support cross data > > > center replication, [....] > > > > Hello, > > > > I'd wager that the question you're really asking about is how to control > > where shards are placed; if you can make deterministic statements about > > where shards are, then you can create your own "rack-aware" or "data > > centre-aware" scenarios. ES has supported this "out of the box" for well > > over a year now (possibly longer). > > > > You'll want to investigate "zones" and "routing allocation", which are > > the key elements of shard placement. There is an excellent blog post which > > describes exactly how to set things up here : > > [http://blog.sematext.com/2012/\*\*05/29/elasticsearch-shard-](http://blog.sematext.com/2012/**05/29/elasticsearch-shard-)\*\* > > placement-control/[http://blog.sematext.com/2012/05/29/elasticsearch-shard-placement-control/](http://blog.sematext.com/2012/05/29/elasticsearch-shard-placement-control/) > > > > Is shard allocation really the correct solution if the data centers are > > globally distributed? > > If I have a data center in the US intended to server data from the US, but > it should also have access to Europe and Asia data, and clusters in both > Europe and Asia with similar needs, would I really want to use zones etc. > and have one great global cluster with data center aware configurations? > > Assuming that the US would be happy to deal with old documents from Asia > and Europe, when Asia or Europe is off line or just not caught up, it would > seem that you would NOT want a "world" cluster, because I can't picture how > you'd configure a 3-part world cluster for both index into the right > indices, search the right (possible combination of) shards, but also > preventing "split brain". > > In the scenerio, I've described, I would think each data center might > better provide availability and eventual consistency (with less concern for > the remote data from the other region) by having three clusters and some > type of syncing from one index to copies at the other two locations. For > example, the US datacenter might have a US, copyOfEurope, and copyOfAsia > index. > > Anyone have any observations about such a world-wide scenerio? > Are there any index to index copy utilities? > Is there a river or other plugin that might be useful for this three > clusters working together scenerio? > How about the project [https://github.com/karussell/\*\*elasticsearch-reindex](https://github.com/karussell/**elasticsearch-reindex)[https://github.com/karussell/elasticsearch-reindex](https://github.com/karussell/elasticsearch-reindex) > ? > Comments? > > -Paul > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com) > . > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > . -- Norberto 'Beto' Meijome -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Todd\_Nine](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/todd_nine/32/778_2.png) [@Todd\_Nine](https://discuss.elastic.co/u/Todd_Nine)\ **Post date:** [June 5, 2014, 4:03am UTC](https://discuss.elastic.co/t/cross-data-center-replication/11670/8 "2014-06-05T04:03:37Z")
Hey all, Sorry to resurrect a dead thread. Did you ever find a solution for eventual consistency of documents across EC2 regions? Thanks, todd On Wednesday, May 1, 2013 5:50:00 AM UTC-7, Norberto Meijome wrote: > +1 on all of the above. es-reindex already in my list of things to > investigate (for a number of issues...) > > cheers, > b > > On Wed, May 1, 2013 at 6:58 AM, Paul Hill \<[pare...@gmail.com](mailto:pare...@gmail.com) \ > > > wrote: > > > On 4/23/2013 8:44 AM, Daniel Maher wrote: > > > > > On 2013-04-23 5:22 PM, Saikat Kanjilal wrote: > > > > > > > Hello Folks, > > > > [...] does ES out of the box currently support cross data > > > > center replication, [....] > > > > > > Hello, > > > > > > I'd wager that the question you're really asking about is how to control > > > where shards are placed; if you can make deterministic statements about > > > where shards are, then you can create your own "rack-aware" or "data > > > centre-aware" scenarios. ES has supported this "out of the box" for well > > > over a year now (possibly longer). > > > > > > You'll want to investigate "zones" and "routing allocation", which are > > > the key elements of shard placement. There is an excellent blog post which > > > describes exactly how to set things up here : > > > [ElasticSearch Shard Placement Control - Sematext](http://blog.sematext.com/2012/05/29/elasticsearch-shard-) > > > placement-control/ > > > > > > Is shard allocation really the correct solution if the data centers are > > > globally distributed? > > > > If I have a data center in the US intended to server data from the US, > > but it should also have access to Europe and Asia data, and clusters in > > both Europe and Asia with similar needs, would I really want to use zones > > etc. and have one great global cluster with data center aware > > configurations? > > > > Assuming that the US would be happy to deal with old documents from Asia > > and Europe, when Asia or Europe is off line or just not caught up, it would > > seem that you would NOT want a "world" cluster, because I can't picture how > > you'd configure a 3-part world cluster for both index into the right > > indices, search the right (possible combination of) shards, but also > > preventing "split brain". > > > > In the scenerio, I've described, I would think each data center might > > better provide availability and eventual consistency (with less concern for > > the remote data from the other region) by having three clusters and some > > type of syncing from one index to copies at the other two locations. For > > example, the US datacenter might have a US, copyOfEurope, and copyOfAsia > > index. > > > > Anyone have any observations about such a world-wide scenerio? > > Are there any index to index copy utilities? > > Is there a river or other plugin that might be useful for this three > > clusters working together scenerio? > > How about the project [GitHub - karussell/elasticsearch-reindex: Simple re-indexing. To backup, apply index settings changes and more ElasticMagic](https://github.com/karussell/elasticsearch-reindex)? > > Comments? > > > > -Paul > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > Norberto 'Beto' Meijome -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/646067d1-1137-4777-be51-ced0bd6a3edd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/646067d1-1137-4777-be51-ced0bd6a3edd%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![MatthewParrott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matthewparrott/32/1454_2.png) [@MatthewParrott](https://discuss.elastic.co/u/MatthewParrott)\ **Post date:** [June 27, 2014, 6:23pm UTC](https://discuss.elastic.co/t/cross-data-center-replication/11670/9 "2014-06-27T18:23:17Z")
I'm interested in this too. es-reindex seems like it lacks conflict resolution, and as noted in the docs, would be better implemented as a river. On Wednesday, June 4, 2014 9:03:37 PM UTC-7, Todd Nine wrote: > Hey all, > > Sorry to resurrect a dead thread. Did you ever find a solution for > eventual consistency of documents across EC2 regions? > > Thanks, > todd > > On Wednesday, May 1, 2013 5:50:00 AM UTC-7, Norberto Meijome wrote: > > > +1 on all of the above. es-reindex already in my list of things to > > investigate (for a number of issues...) > > > > cheers, > > b > > > > On Wed, May 1, 2013 at 6:58 AM, Paul Hill [pare...@gmail.com](mailto:pare...@gmail.com) wrote: > > > > > On 4/23/2013 8:44 AM, Daniel Maher wrote: > > > > > > > On 2013-04-23 5:22 PM, Saikat Kanjilal wrote: > > > > > > > > > Hello Folks, > > > > > [...] does ES out of the box currently support cross data > > > > > center replication, [....] > > > > > > > > Hello, > > > > > > > > I'd wager that the question you're really asking about is how to > > > > control where shards are placed; if you can make deterministic statements > > > > about where shards are, then you can create your own "rack-aware" or "data > > > > centre-aware" scenarios. ES has supported this "out of the box" for well > > > > over a year now (possibly longer). > > > > > > > > You'll want to investigate "zones" and "routing allocation", which are > > > > the key elements of shard placement. There is an excellent blog post which > > > > describes exactly how to set things up here : > > > > [ElasticSearch Shard Placement Control - Sematext](http://blog.sematext.com/2012/05/29/elasticsearch-shard-) > > > > placement-control/ > > > > > > > > Is shard allocation really the correct solution if the data centers > > > > are globally distributed? > > > > > > If I have a data center in the US intended to server data from the US, > > > but it should also have access to Europe and Asia data, and clusters in > > > both Europe and Asia with similar needs, would I really want to use zones > > > etc. and have one great global cluster with data center aware > > > configurations? > > > > > > Assuming that the US would be happy to deal with old documents from Asia > > > and Europe, when Asia or Europe is off line or just not caught up, it would > > > seem that you would NOT want a "world" cluster, because I can't picture how > > > you'd configure a 3-part world cluster for both index into the right > > > indices, search the right (possible combination of) shards, but also > > > preventing "split brain". > > > > > > In the scenerio, I've described, I would think each data center might > > > better provide availability and eventual consistency (with less concern for > > > the remote data from the other region) by having three clusters and some > > > type of syncing from one index to copies at the other two locations. For > > > example, the US datacenter might have a US, copyOfEurope, and copyOfAsia > > > index. > > > > > > Anyone have any observations about such a world-wide scenerio? > > > Are there any index to index copy utilities? > > > Is there a river or other plugin that might be useful for this three > > > clusters working together scenerio? > > > How about the project [GitHub - karussell/elasticsearch-reindex: Simple re-indexing. To backup, apply index settings changes and more ElasticMagic](https://github.com/karussell/elasticsearch-reindex) > > > ? > > > Comments? > > > > > > -Paul > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > Norberto 'Beto' Meijome -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/86f03167-6803-4bdd-9278-21b222e56d7c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/86f03167-6803-4bdd-9278-21b222e56d7c%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:19am UTC](https://discuss.elastic.co/t/cross-data-center-replication/11670/10 "2017-07-06T01:19:14Z")