# How to setup an ES-to-ES river? **URL:** \ **Category:** Elasticsearch\ **Created:** [September 28, 2012, 12:49am UTC](https://discuss.elastic.co/t/how-to-setup-an-es-to-es-river/9182 "2012-09-28T00:49:56Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![es\_learner](https://avatars.discourse-cdn.com/v4/letter/e/8dc957/32.png) [@es\_learner](https://discuss.elastic.co/u/es_learner)\ **Post date:** [September 28, 2012, 12:49am UTC](https://discuss.elastic.co/t/how-to-setup-an-es-to-es-river/9182/1 "2012-09-28T00:49:56Z")
Is it supported? The objective here is to 'tee' new docs into a secondary index. My current implementation is to write twice from the client - once to primary index and the other to secondary. Primary index gets pruned every month. Secondary is never pruned. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 28, 2012, 1:06am UTC](https://discuss.elastic.co/t/how-to-setup-an-es-to-es-river/9182/2 "2012-09-28T01:06:12Z")
See here: [https://github.com/elasticsearch/elasticsearch/issues/1077](https://github.com/elasticsearch/elasticsearch/issues/1077) -- David 馃槈 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 28 sept. 2012 脿 02:49, es\_learner [dave@livefyre.com](mailto:dave@livefyre.com) a 茅crit : Is it supported? The objective here is to 'tee' new docs into a secondary index. My current implementation is to write twice from the client - once to primary index and the other to secondary. Primary index gets pruned every month. Secondary is never pruned. -- View this message in context: [http://elasticsearch-users.115913.n3.nabble.com/How-to-setup-an-ES-to-ES-river-tp4023286.html](http://elasticsearch-users.115913.n3.nabble.com/How-to-setup-an-ES-to-ES-river-tp4023286.html) Sent from the ElasticSearch Users mailing list archive at [Nabble.com](http://Nabble.com). -- -- ---
**Author:** ![James\_Boehmer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_boehmer/32/2636_2.png) [@James\_Boehmer](https://discuss.elastic.co/u/James_Boehmer)\ **Post date:** [November 8, 2012, 7:41pm UTC](https://discuss.elastic.co/t/how-to-setup-an-es-to-es-river/9182/3 "2012-11-08T19:41:35Z")
I'm actually looking for something very similar, which I do not believe is the same as that \_source river request. I need to run two Elasticsearch stacks separately but simultaneously, to segregate internal traffic from external traffic. With Solr I would set up a single master, and run two sets of slaves load balanced independently. That way the internal slaves could never be affected by traffic hitting the external slaves, and vice versa. But with ES is there a way to set up a handful of nodes that are basically their own cluster, but get their data from a master cluster which does not store the \_source? On Thursday, September 27, 2012 9:06:17 PM UTC-4, David Pilato wrote: > See here: [[Feature Request] Add a river to ElasticSearch instance 路 Issue #1077 路 elastic/elasticsearch 路 GitHub](https://github.com/elasticsearch/elasticsearch/issues/1077) > > -- > David 馃槈 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 28 sept. 2012 脿 02:49, es\_learner \<[da...@livefyre.com](mailto:da...@livefyre.com) \\> a > 茅crit : > > Is it supported? > > The objective here is to 'tee' new docs into a secondary index. My current > implementation is to write twice from the client - once to primary index > and > the other to secondary. Primary index gets pruned every month. Secondary > is never pruned. > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/How-to-setup-an-ES-to-ES-river-tp4023286.html](http://elasticsearch-users.115913.n3.nabble.com/How-to-setup-an-ES-to-ES-river-tp4023286.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > -- -- ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [November 9, 2012, 12:53am UTC](https://discuss.elastic.co/t/how-to-setup-an-es-to-es-river/9182/4 "2012-11-09T00:53:05Z")
Hi, can you please elaborate what is the kind of "traffic"? Is it data load for indexing, or search requests hitting the cluster, or both of them? You can set up data-less ES nodes that can absorb all the network connection load, that is very easy. You can also dedicate data-less nodes to different ports, if you mean that by addressing internal/external traffic. J枚rg On Thursday, November 8, 2012 8:41:35 PM UTC+1, James Boehmer wrote: > I'm actually looking for something very similar, which I do not believe is > the same as that \_source river request. I need to run two Elasticsearch > stacks separately but simultaneously, to segregate internal traffic from > external traffic. With Solr I would set up a single master, and run two > sets of slaves load balanced independently. That way the internal slaves > could never be affected by traffic hitting the external slaves, and vice > versa. But with ES is there a way to set up a handful of nodes that are > basically their own cluster, but get their data from a master cluster which > does not store the \_source? > > On Thursday, September 27, 2012 9:06:17 PM UTC-4, David Pilato wrote: > > > See here: [[Feature Request] Add a river to ElasticSearch instance 路 Issue #1077 路 elastic/elasticsearch 路 GitHub](https://github.com/elasticsearch/elasticsearch/issues/1077) > > > > -- > > David 馃槈 > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > Le 28 sept. 2012 脿 02:49, es\_learner [da...@livefyre.com](mailto:da...@livefyre.com) a 茅crit : > > > > Is it supported? > > > > The objective here is to 'tee' new docs into a secondary index. My > > current > > implementation is to write twice from the client - once to primary index > > and > > the other to secondary. Primary index gets pruned every month. Secondary > > is never pruned. > > > > -- > > View this message in context: > > [http://elasticsearch-users.115913.n3.nabble.com/How-to-setup-an-ES-to-ES-river-tp4023286.html](http://elasticsearch-users.115913.n3.nabble.com/How-to-setup-an-ES-to-ES-river-tp4023286.html) > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > > > -- -- ---
**Author:** ![James\_Boehmer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_boehmer/32/2636_2.png) [@James\_Boehmer](https://discuss.elastic.co/u/James_Boehmer)\ **Post date:** [November 9, 2012, 2:15am UTC](https://discuss.elastic.co/t/how-to-setup-an-es-to-es-river/9182/5 "2012-11-09T02:15:39Z")
It would be solely for querying. For example, we'd like to have a cluster with 5 shards/1 replica being constantly indexed and queried. Then we'd like to have a second cluster for serving external query traffic, but would get its data from the first cluster. The second cluster would have its own complete set of primary/replica shards separate from the first cluster. However, we would like it to index it passively from the first cluster instead of having to manually index both clusters simultaneously. The purpose of the second cluster is to be able to scale and absorb traffic independently from the internal cluster. It's somewhat important that they not interfere with each other, but I suppose that an entire single cluster could scale to handle all of the traffic anyway. On Thursday, November 8, 2012 7:53:05 PM UTC-5, J枚rg Prante wrote: > Hi, > > can you please elaborate what is the kind of "traffic"? Is it data load > for indexing, or search requests hitting the cluster, or both of them? > You can set up data-less ES nodes that can absorb all the network > connection load, that is very easy. You can also dedicate data-less nodes > to different ports, if you mean that by addressing internal/external > traffic. > > J枚rg > > On Thursday, November 8, 2012 8:41:35 PM UTC+1, James Boehmer wrote: > > > I'm actually looking for something very similar, which I do not believe > > is the same as that \_source river request. I need to run two Elasticsearch > > stacks separately but simultaneously, to segregate internal traffic from > > external traffic. With Solr I would set up a single master, and run two > > sets of slaves load balanced independently. That way the internal slaves > > could never be affected by traffic hitting the external slaves, and vice > > versa. But with ES is there a way to set up a handful of nodes that are > > basically their own cluster, but get their data from a master cluster which > > does not store the \_source? > > > > On Thursday, September 27, 2012 9:06:17 PM UTC-4, David Pilato wrote: > > > > > See here: [[Feature Request] Add a river to ElasticSearch instance 路 Issue #1077 路 elastic/elasticsearch 路 GitHub](https://github.com/elasticsearch/elasticsearch/issues/1077) > > > > > > -- > > > David 馃槈 > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > Le 28 sept. 2012 脿 02:49, es\_learner [da...@livefyre.com](mailto:da...@livefyre.com) a 茅crit : > > > > > > Is it supported? > > > > > > The objective here is to 'tee' new docs into a secondary index. My > > > current > > > implementation is to write twice from the client - once to primary index > > > and > > > the other to secondary. Primary index gets pruned every month. > > > Secondary > > > is never pruned. > > > > > > -- > > > View this message in context: > > > [http://elasticsearch-users.115913.n3.nabble.com/How-to-setup-an-ES-to-ES-river-tp4023286.html](http://elasticsearch-users.115913.n3.nabble.com/How-to-setup-an-ES-to-ES-river-tp4023286.html) > > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > > > > > -- -- ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [November 9, 2012, 10:37am UTC](https://discuss.elastic.co/t/how-to-setup-an-es-to-es-river/9182/6 "2012-11-09T10:37:13Z")
Hi James, you can choose a setup within a single cluster, where the nodes (the cluster members) serve different purposes. No need for a second cluster. ES nodes can be started in a data-only mode, without HTTP server, so they never process client requests, but only do the heavy lifting. Proxy nodes can be started without data, but with HTTP, so they only process client requests and forward them to the data nodes involved in the queries. You can start as many proxy nodes and data nodes as you want, so you scale the nodes in two aspects. In my view, if you separate proxy and data nodes into two clusters, there are much hassles. Nodes can not talk to each other over cluster boundaries. You would have to store your data twice by doing it with your client tool alone (while ES can do it for you a lot easier by using replica levels), and afterwards, you would have to keep the data in sync when nodes fail (what is tedious when doing it with external client tools, while ES is doing it for you automatically by replicated shards and allocation control). Cheers, J枚rg On Friday, November 9, 2012 3:15:39 AM UTC+1, James Boehmer wrote: > It would be solely for querying. For example, we'd like to have a cluster > with 5 shards/1 replica being constantly indexed and queried. Then we'd > like to have a second cluster for serving external query traffic, but would > get its data from the first cluster. The second cluster would have its own > complete set of primary/replica shards separate from the first cluster. > However, we would like it to index it passively from the first cluster > instead of having to manually index both clusters simultaneously. The > purpose of the second cluster is to be able to scale and absorb traffic > independently from the internal cluster. It's somewhat important that they > not interfere with each other, but I suppose that an entire single cluster > could scale to handle all of the traffic anyway. > > On Thursday, November 8, 2012 7:53:05 PM UTC-5, J枚rg Prante wrote: > > > Hi, > > > > can you please elaborate what is the kind of "traffic"? Is it data load > > for indexing, or search requests hitting the cluster, or both of them? > > You can set up data-less ES nodes that can absorb all the network > > connection load, that is very easy. You can also dedicate data-less nodes > > to different ports, if you mean that by addressing internal/external > > traffic. > > > > J枚rg > > > > On Thursday, November 8, 2012 8:41:35 PM UTC+1, James Boehmer wrote: > > > > > I'm actually looking for something very similar, which I do not believe > > > is the same as that \_source river request. I need to run two Elasticsearch > > > stacks separately but simultaneously, to segregate internal traffic from > > > external traffic. With Solr I would set up a single master, and run two > > > sets of slaves load balanced independently. That way the internal slaves > > > could never be affected by traffic hitting the external slaves, and vice > > > versa. But with ES is there a way to set up a handful of nodes that are > > > basically their own cluster, but get their data from a master cluster which > > > does not store the \_source? > > > > > > On Thursday, September 27, 2012 9:06:17 PM UTC-4, David Pilato wrote: > > > > > > > See here: [[Feature Request] Add a river to ElasticSearch instance 路 Issue #1077 路 elastic/elasticsearch 路 GitHub](https://github.com/elasticsearch/elasticsearch/issues/1077) > > > > > > > > -- > > > > David 馃槈 > > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > > > Le 28 sept. 2012 脿 02:49, es\_learner [da...@livefyre.com](mailto:da...@livefyre.com) a 茅crit : > > > > > > > > Is it supported? > > > > > > > > The objective here is to 'tee' new docs into a secondary index. My > > > > current > > > > implementation is to write twice from the client - once to primary > > > > index and > > > > the other to secondary. Primary index gets pruned every month. > > > > Secondary > > > > is never pruned. > > > > > > > > -- > > > > View this message in context: > > > > [http://elasticsearch-users.115913.n3.nabble.com/How-to-setup-an-ES-to-ES-river-tp4023286.html](http://elasticsearch-users.115913.n3.nabble.com/How-to-setup-an-ES-to-ES-river-tp4023286.html) > > > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > > > > > > > -- -- ---
**Author:** ![James\_Boehmer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_boehmer/32/2636_2.png) [@James\_Boehmer](https://discuss.elastic.co/u/James_Boehmer)\ **Post date:** [November 10, 2012, 1:29pm UTC](https://discuss.elastic.co/t/how-to-setup-an-es-to-es-river/9182/7 "2012-11-10T13:29:48Z")
Hi J枚rg, We would like each cluster to do its own heavy lifting. As for HTTP, we are using a load balancer for insertions and queries, so essentially every node in the clusters gets to serve both purposes in round robin fashion. I do not think separating the HTTP requests from the heavy lifting of searching is quite what we're looking for in this situation. But what do you mean by replica levels? Would that imply creating additional replicas of a shard, and assigning them to specific nodes? -Jim On Friday, November 9, 2012 5:37:14 AM UTC-5, J枚rg Prante wrote: > Hi James, > > you can choose a setup within a single cluster, where the nodes (the > cluster members) serve different purposes. No need for a second cluster. > > ES nodes can be started in a data-only mode, without HTTP server, so they > never process client requests, but only do the heavy lifting. > > Proxy nodes can be started without data, but with HTTP, so they only > process client requests and forward them to the data nodes involved in the > queries. > > You can start as many proxy nodes and data nodes as you want, so you scale > the nodes in two aspects. > > In my view, if you separate proxy and data nodes into two clusters, there > are much hassles. Nodes can not talk to each other over cluster boundaries. > You would have to store your data twice by doing it with your client tool > alone (while ES can do it for you a lot easier by using replica levels), > and afterwards, you would have to keep the data in sync when nodes fail > (what is tedious when doing it with external client tools, while ES is > doing it for you automatically by replicated shards and allocation control). > > Cheers, > > J枚rg > > On Friday, November 9, 2012 3:15:39 AM UTC+1, James Boehmer wrote: > > > It would be solely for querying. For example, we'd like to have a > > cluster with 5 shards/1 replica being constantly indexed and queried. Then > > we'd like to have a second cluster for serving external query traffic, but > > would get its data from the first cluster. The second cluster would have > > its own complete set of primary/replica shards separate from the first > > cluster. However, we would like it to index it passively from the first > > cluster instead of having to manually index both clusters simultaneously. > > The purpose of the second cluster is to be able to scale and absorb > > traffic independently from the internal cluster. It's somewhat important > > that they not interfere with each other, but I suppose that an entire > > single cluster could scale to handle all of the traffic anyway. > > > > On Thursday, November 8, 2012 7:53:05 PM UTC-5, J枚rg Prante wrote: > > > > > Hi, > > > > > > can you please elaborate what is the kind of "traffic"? Is it data load > > > for indexing, or search requests hitting the cluster, or both of them? > > > You can set up data-less ES nodes that can absorb all the network > > > connection load, that is very easy. You can also dedicate data-less nodes > > > to different ports, if you mean that by addressing internal/external > > > traffic. > > > > > > J枚rg > > > > > > On Thursday, November 8, 2012 8:41:35 PM UTC+1, James Boehmer wrote: > > > > > > > I'm actually looking for something very similar, which I do not believe > > > > is the same as that \_source river request. I need to run two Elasticsearch > > > > stacks separately but simultaneously, to segregate internal traffic from > > > > external traffic. With Solr I would set up a single master, and run two > > > > sets of slaves load balanced independently. That way the internal slaves > > > > could never be affected by traffic hitting the external slaves, and vice > > > > versa. But with ES is there a way to set up a handful of nodes that are > > > > basically their own cluster, but get their data from a master cluster which > > > > does not store the \_source? > > > > > > > > On Thursday, September 27, 2012 9:06:17 PM UTC-4, David Pilato wrote: > > > > > > > > > See here: [[Feature Request] Add a river to ElasticSearch instance 路 Issue #1077 路 elastic/elasticsearch 路 GitHub](https://github.com/elasticsearch/elasticsearch/issues/1077) > > > > > > > > > > -- > > > > > David 馃槈 > > > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > > > > > Le 28 sept. 2012 脿 02:49, es\_learner [da...@livefyre.com](mailto:da...@livefyre.com) a 茅crit : > > > > > > > > > > Is it supported? > > > > > > > > > > The objective here is to 'tee' new docs into a secondary index. My > > > > > current > > > > > implementation is to write twice from the client - once to primary > > > > > index and > > > > > the other to secondary. Primary index gets pruned every month. > > > > > Secondary > > > > > is never pruned. > > > > > > > > > > -- > > > > > View this message in context: > > > > > [http://elasticsearch-users.115913.n3.nabble.com/How-to-setup-an-ES-to-ES-river-tp4023286.html](http://elasticsearch-users.115913.n3.nabble.com/How-to-setup-an-ES-to-ES-river-tp4023286.html) > > > > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > > > > > > > > > -- -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:04am UTC](https://discuss.elastic.co/t/how-to-setup-an-es-to-es-river/9182/8 "2017-07-06T03:04:58Z")