# Replicating one cluster to another cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [January 8, 2014, 12:26am UTC](https://discuss.elastic.co/t/replicating-one-cluster-to-another-cluster/15132 "2014-01-08T00:26:23Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Zuhaib\_Siddique](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zuhaib_siddique/32/1896_2.png) [@Zuhaib\_Siddique](https://discuss.elastic.co/u/Zuhaib_Siddique)\ **Post date:** [January 8, 2014, 12:26am UTC](https://discuss.elastic.co/t/replicating-one-cluster-to-another-cluster/15132/1 "2014-01-08T00:26:23Z")
Hey, Was wondering if anyone has tried replicating one cluster to a new cluster and keep it in "sync". Example is I have a production cluster and i need to reindex all data. I would like to do this in a 2nd cluster so I can compare the changes but if an update happens on the original index I want it reflected on the replicated one. I am pretty sure I can whip something with scroll/scan but if someone has done before and has code to share it would be great. Thanks Zuhaib -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c9f39a38-53f4-4f23-9aaa-0b270261ebae%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c9f39a38-53f4-4f23-9aaa-0b270261ebae%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 8, 2014, 1:01am UTC](https://discuss.elastic.co/t/replicating-one-cluster-to-another-cluster/15132/2 "2014-01-08T01:01:36Z")
There are a few ways; stream2es - [GitHub - elastic/stream2es: Stream data into ES (Wikipedia, Twitter, stdin, or other ESes)](https://github.com/elasticsearch/stream2es) logstash with the elasticsearch and elasticsearch\_http outputs - [http://logstash.net/](http://logstash.net/) There is also these two which I haven't used - [GitHub - crate/elasticsearch-inout-plugin: An Elasticsearch plugin which provides the ability to export data by query on server side.](https://github.com/crate/elasticsearch-inout-plugin) And - [GitHub - jprante/elasticsearch-knapsack: Knapsack plugin is an import/export tool for Elasticsearch](https://github.com/jprante/elasticsearch-knapsack) Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 8 January 2014 11:26, Zuhaib Siddique [zsiddique@gmail.com](mailto:zsiddique@gmail.com) wrote: > Hey, > > Was wondering if anyone has tried replicating one cluster to a new cluster > and keep it in "sync". Example is I have a production cluster and i need > to reindex all data. I would like to do this in a 2nd cluster so I can > compare the changes but if an update happens on the original index I want > it reflected on the replicated one. > > I am pretty sure I can whip something with scroll/scan but if someone has > done before and has code to share it would be great. > > Thanks > Zuhaib > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/c9f39a38-53f4-4f23-9aaa-0b270261ebae%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c9f39a38-53f4-4f23-9aaa-0b270261ebae%40googlegroups.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624a9XG2jd-%2BO4TvxCJ\_v6odn%3DoeG9aiJNiQsPsa8CSYk9g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624a9XG2jd-%2BO4TvxCJ_v6odn%3DoeG9aiJNiQsPsa8CSYk9g%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [January 8, 2014, 9:14am UTC](https://discuss.elastic.co/t/replicating-one-cluster-to-another-cluster/15132/3 "2014-01-08T09:14:31Z")
First and most important, the good news: ES 1.0.0.Beta2 has snapshot/restore feature in place so it should be easy to snapshot and restore the result back to a target cluster. The snapshots are also incremental. Second, there are also news for the knapsack plugin. In the next knapsack plugin version due this week, a full copy from cluster1 to cluster2 will be as simple as curl -XPOST '[http://cluster1node](http://cluster1node):port1 /\_export/copy?cluster=cluster2name&host=cluster2node&port=port2' Limitations will be that you have knapsack plugin installed at cluster1node, the same JVM version in cluster1 and cluster2, same ES version in cluster1 and cluster2, and all your indexes have stored fields, preferably the \_source field. Also, cluster1 must not modify the indexes while the \_export/copy is running, or cluster2 may have different data (there is no inherent locking). In the new knapsack export version, you will be able to use arbitrary ES queries to select subsets of the cluster data to copy, so only the hits of a query can be transferred. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHn%2BLA-BHeZTzxr6C2w4g7ULqWLHpr6gw6zstWptmDt4g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHn%2BLA-BHeZTzxr6C2w4g7ULqWLHpr6gw6zstWptmDt4g%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![abinns](https://avatars.discourse-cdn.com/v4/letter/a/3be4f8/32.png) [@abinns](https://discuss.elastic.co/u/abinns)\ **Post date:** [January 8, 2014, 6:17pm UTC](https://discuss.elastic.co/t/replicating-one-cluster-to-another-cluster/15132/4 "2014-01-08T18:17:11Z")
A colleague of mine here at TaskRabbit whipped-up a node.js-based tool similar to these: > **[taskrabbit/elasticsearch-dump](https://github.com/taskrabbit/elasticsearch-dump)** > > elasticsearch-dump - Import and export tools for elasticsearch His main use case is to replicate ES cluster from our production system to a staging/test environment. I believe it has the same requirements as other similar tools, mainly that the source index needs to have the original documents stored in the \_source field. Feedback and criticism is welcome. Aaron -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8a7601f6-8484-40c2-9576-9e6c19dcc7d3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8a7601f6-8484-40c2-9576-9e6c19dcc7d3%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:57am UTC](https://discuss.elastic.co/t/replicating-one-cluster-to-another-cluster/15132/5 "2017-07-06T01:57:53Z")