# Queries regarding the rivers implementation **URL:** \ **Category:** Elasticsearch\ **Created:** [September 29, 2010, 6:22am UTC](https://discuss.elastic.co/t/queries-regarding-the-rivers-implementation/3383 "2010-09-29T06:22:47Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Mahendra\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mahendra_m/32/3128_2.png) [@Mahendra\_M](https://discuss.elastic.co/u/Mahendra_M)\ **Post date:** [September 29, 2010, 6:22am UTC](https://discuss.elastic.co/t/queries-regarding-the-rivers-implementation/3383/1 "2010-09-29T06:22:47Z")
Hi, I was going through the elasticsearch code and docs (couchdb river) and had some doubts. (I am still in the process of setting up ES 0.11). 1. The couchdb river plugin provides extra settings for "index" [1]. Does this provide support for creating a index other than "\_river" ? From the rivers documentation [2], it looks like the index name will be \_river and the user can control only the type. Or is it actually possible to override \_river with a different index name ? 2. CouchDB filter args - CouchDB supports arguments to filters [3], which can be useful in filtering out specific data. From the code, it looks like we can add this as part of 'filter' configuration "filter" = "abc&option1=value1" In the code for CouchDB river, it is just if (couchFilter != null) { file = file + "&filter=" + couchFilter; } So, the HTTP url will get formed as '....&filter=abc&option1=value1', but it looks like a hack. Would it be better to provide a different configuration for filter parameters or is the above hack a safe way to do things ? [1] [http://www.elasticsearch.com/docs/elasticsearch/river/couchdb/](http://www.elasticsearch.com/docs/elasticsearch/river/couchdb/) [2] [http://www.elasticsearch.com/blog/2010/09/28/the\_river.html](http://www.elasticsearch.com/blog/2010/09/28/the_river.html) [3] [http://blog.couchone.com/post/468392274/whats-new-in-apache-couchdb-0-11-part-three-new](http://blog.couchone.com/post/468392274/whats-new-in-apache-couchdb-0-11-part-three-new) Regards, Mahendra [http://twitter.com/mahendra](http://twitter.com/mahendra) ---
**Author:** ![Mahendra\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mahendra_m/32/3128_2.png) [@Mahendra\_M](https://discuss.elastic.co/u/Mahendra_M)\ **Post date:** [September 29, 2010, 8:29am UTC](https://discuss.elastic.co/t/queries-regarding-the-rivers-implementation/3383/2 "2010-09-29T08:29:07Z")
Hi, I guess I found an answer to one of my own questions. On Wed, Sep 29, 2010 at 11:52 AM, Mahendra M wrote: > 1. The couchdb river plugin provides extra settings for "index" [1]. > Does this provide support for creating a index other than "\_river" ? > From the rivers documentation [2], it looks like the index name will > be \_river and the user can control only the type. Or is it actually > possible to override \_river with a different index name ? From trying it out, I think the interpretation for the index settings is as follows. (Please correct me if I am wrong) While creating a couchdb river, if "index" is not specified, then the default values are index = \ type = \ So, searching for data has to be done against : [http://localhost:9200/](http://localhost:9200/)\/\/\_search I was (wrongly) assuming that the index will be created as index = \_river type = \ and search query had to be done using: [http://localhost:9200/\_river/](http://localhost:9200/_river/)\/\_search Also, if "index" and "type" parameters are specified while setting up the river, then Elasticsearch will use those instead of \. Sorry for the confusion. I could not figure it out from the documentation. Regards, Mahendra [http://twitter.com/mahendra](http://twitter.com/mahendra) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 29, 2010, 9:33am UTC](https://discuss.elastic.co/t/queries-regarding-the-rivers-implementation/3383/3 "2010-09-29T09:33:17Z")
Hey, Yea, the \_river index acts as the registry for rivers and their metadata and state (providing persistency and state management for the rivers, not the data they index). For example, the coucdb one stores the last indexed seq in it. You can control the index name and type you index to. Regarding the filter, I misses the part where you can provide parameters to filters, I can add it as a configuration option. Want to open an issue for it? -shay.banon On Wed, Sep 29, 2010 at 10:29 AM, Mahendra M [mahendra.m@gmail.com](mailto:mahendra.m@gmail.com) wrote: > Hi, > > I guess I found an answer to one of my own questions. > > On Wed, Sep 29, 2010 at 11:52 AM, Mahendra M wrote: > > > 1. The couchdb river plugin provides extra settings for "index" [1]. > > Does this provide support for creating a index other than "\_river" ? > > From the rivers documentation [2], it looks like the index name will > > be \_river and the user can control only the type. Or is it actually > > possible to override \_river with a different index name ? > > From trying it out, I think the interpretation for the index settings > is as follows. (Please correct me if I am wrong) > > While creating a couchdb river, if "index" is not specified, then the > default values are > index = \ > type = \ > > So, searching for data has to be done against : > [http://localhost:9200/](http://localhost:9200/)\/\/\_search > > I was (wrongly) assuming that the index will be created as > index = \_river > type = \ > > and search query had to be done using: > [http://localhost:9200/\_river/](http://localhost:9200/_river/)\/\_search > > Also, if "index" and "type" parameters are specified while setting up > the river, then Elasticsearch will use those instead of \. > > Sorry for the confusion. I could not figure it out from the documentation. > > Regards, > Mahendra > > [http://twitter.com/mahendra](http://twitter.com/mahendra) ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [September 29, 2010, 10:10am UTC](https://discuss.elastic.co/t/queries-regarding-the-rivers-implementation/3383/4 "2010-09-29T10:10:59Z")
Just out of curiosity, how it is possible to control the index and type name for RabbitMQ river? Also I found that it is possible to customize the river registry name by settings river.index\_name property (defaults to "\_river"). Probably should be added to the documentation. Lukas On Wed, Sep 29, 2010 at 11:33 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Hey, > > Yea, the \_river index acts as the registry for rivers and their metadata > and state (providing persistency and state management for the rivers, not > the data they index). For example, the coucdb one stores the last indexed > seq in it. You can control the index name and type you index to. > > Regarding the filter, I misses the part where you can provide parameters > to filters, I can add it as a configuration option. Want to open an issue > for it? > > -shay.banon > > On Wed, Sep 29, 2010 at 10:29 AM, Mahendra M [mahendra.m@gmail.com](mailto:mahendra.m@gmail.com) wrote: > > > Hi, > > > > I guess I found an answer to one of my own questions. > > > > On Wed, Sep 29, 2010 at 11:52 AM, Mahendra M wrote: > > > > > 1. The couchdb river plugin provides extra settings for "index" [1]. > > > Does this provide support for creating a index other than "\_river" ? > > > From the rivers documentation [2], it looks like the index name will > > > be \_river and the user can control only the type. Or is it actually > > > possible to override \_river with a different index name ? > > > > From trying it out, I think the interpretation for the index settings > > is as follows. (Please correct me if I am wrong) > > > > While creating a couchdb river, if "index" is not specified, then the > > default values are > > index = \ > > type = \ > > > > So, searching for data has to be done against : > > [http://localhost:9200/](http://localhost:9200/)\/\/\_search > > > > I was (wrongly) assuming that the index will be created as > > index = \_river > > type = \ > > > > and search query had to be done using: > > [http://localhost:9200/\_river/](http://localhost:9200/_river/)\/\_search > > > > Also, if "index" and "type" parameters are specified while setting up > > the river, then Elasticsearch will use those instead of \. > > > > Sorry for the confusion. I could not figure it out from the documentation. > > > > Regards, > > Mahendra > > > > [http://twitter.com/mahendra](http://twitter.com/mahendra) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 29, 2010, 10:19am UTC](https://discuss.elastic.co/t/queries-regarding-the-rivers-implementation/3383/5 "2010-09-29T10:19:09Z")
The rabbitmq has supports the bulk format, so all information (index, type) is provided per bulk item. -shay.banon On Wed, Sep 29, 2010 at 12:10 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com) wrote: > Just out of curiosity, how it is possible to control the index and type > name for RabbitMQ river? > > Also I found that it is possible to customize the river registry name by > settings river.index\_name property (defaults to "\_river"). Probably should > be added to the documentation. > > Lukas > > On Wed, Sep 29, 2010 at 11:33 AM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) > > > wrote: > > > Hey, > > > > Yea, the \_river index acts as the registry for rivers and their > > metadata and state (providing persistency and state management for the > > rivers, not the data they index). For example, the coucdb one stores the > > last indexed seq in it. You can control the index name and type you index > > to. > > > > Regarding the filter, I misses the part where you can provide > > parameters to filters, I can add it as a configuration option. Want to open > > an issue for it? > > > > -shay.banon > > > > On Wed, Sep 29, 2010 at 10:29 AM, Mahendra M [mahendra.m@gmail.com](mailto:mahendra.m@gmail.com)wrote: > > > > > Hi, > > > > > > I guess I found an answer to one of my own questions. > > > > > > On Wed, Sep 29, 2010 at 11:52 AM, Mahendra M wrote: > > > > > > > 1. The couchdb river plugin provides extra settings for "index" [1]. > > > > Does this provide support for creating a index other than "\_river" ? > > > > From the rivers documentation [2], it looks like the index name will > > > > be \_river and the user can control only the type. Or is it actually > > > > possible to override \_river with a different index name ? > > > > > > From trying it out, I think the interpretation for the index settings > > > is as follows. (Please correct me if I am wrong) > > > > > > While creating a couchdb river, if "index" is not specified, then the > > > default values are > > > index = \ > > > type = \ > > > > > > So, searching for data has to be done against : > > > [http://localhost:9200/](http://localhost:9200/)\/\/\_search > > > > > > I was (wrongly) assuming that the index will be created as > > > index = \_river > > > type = \ > > > > > > and search query had to be done using: > > > [http://localhost:9200/\_river/](http://localhost:9200/_river/)\/\_search > > > > > > Also, if "index" and "type" parameters are specified while setting up > > > the river, then Elasticsearch will use those instead of \. > > > > > > Sorry for the confusion. I could not figure it out from the > > > documentation. > > > > > > Regards, > > > Mahendra > > > > > > [http://twitter.com/mahendra](http://twitter.com/mahendra) ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [September 29, 2010, 10:21am UTC](https://discuss.elastic.co/t/queries-regarding-the-rivers-implementation/3383/6 "2010-09-29T10:21:04Z")
Ah... yea... thnx. On Wed, Sep 29, 2010 at 12:19 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > The rabbitmq has supports the bulk format, so all information (index, type) > is provided per bulk item. > > -shay.banon > > On Wed, Sep 29, 2010 at 12:10 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote: > > > Just out of curiosity, how it is possible to control the index and type > > name for RabbitMQ river? > > > > Also I found that it is possible to customize the river registry name by > > settings river.index\_name property (defaults to "\_river"). Probably should > > be added to the documentation. > > > > Lukas > > > > On Wed, Sep 29, 2010 at 11:33 AM, Shay Banon \< > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote: > > > > > Hey, > > > > > > Yea, the \_river index acts as the registry for rivers and their > > > metadata and state (providing persistency and state management for the > > > rivers, not the data they index). For example, the coucdb one stores the > > > last indexed seq in it. You can control the index name and type you index > > > to. > > > > > > Regarding the filter, I misses the part where you can provide > > > parameters to filters, I can add it as a configuration option. Want to open > > > an issue for it? > > > > > > -shay.banon > > > > > > On Wed, Sep 29, 2010 at 10:29 AM, Mahendra M [mahendra.m@gmail.com](mailto:mahendra.m@gmail.com)wrote: > > > > > > > Hi, > > > > > > > > I guess I found an answer to one of my own questions. > > > > > > > > On Wed, Sep 29, 2010 at 11:52 AM, Mahendra M wrote: > > > > > > > > > 1. The couchdb river plugin provides extra settings for "index" [1]. > > > > > Does this provide support for creating a index other than "\_river" ? > > > > > From the rivers documentation [2], it looks like the index name will > > > > > be \_river and the user can control only the type. Or is it actually > > > > > possible to override \_river with a different index name ? > > > > > > > > From trying it out, I think the interpretation for the index settings > > > > is as follows. (Please correct me if I am wrong) > > > > > > > > While creating a couchdb river, if "index" is not specified, then the > > > > default values are > > > > index = \ > > > > type = \ > > > > > > > > So, searching for data has to be done against : > > > > [http://localhost:9200/](http://localhost:9200/)\/\/\_search > > > > > > > > I was (wrongly) assuming that the index will be created as > > > > index = \_river > > > > type = \ > > > > > > > > and search query had to be done using: > > > > [http://localhost:9200/\_river/](http://localhost:9200/_river/)\/\_search > > > > > > > > Also, if "index" and "type" parameters are specified while setting up > > > > the river, then Elasticsearch will use those instead of \. > > > > > > > > Sorry for the confusion. I could not figure it out from the > > > > documentation. > > > > > > > > Regards, > > > > Mahendra > > > > > > > > [http://twitter.com/mahendra](http://twitter.com/mahendra) ---
**Author:** ![Mahendra\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mahendra_m/32/3128_2.png) [@Mahendra\_M](https://discuss.elastic.co/u/Mahendra_M)\ **Post date:** [September 29, 2010, 10:27am UTC](https://discuss.elastic.co/t/queries-regarding-the-rivers-implementation/3383/7 "2010-09-29T10:27:03Z")
Hi Shay, On Wed, Sep 29, 2010 at 3:03 PM, Shay Banon wrote: > Yea, the \_river index acts as the registry for rivers and their metadata > and state (providing persistency and state management for the rivers, not > the data they index). For example, the coucdb one stores the last indexed > seq in it. You can control the index name and type you index to. OK cool. So, I guess if one has to upload a mapping for an index (river), it must be on the actual index and not the \_river index. > Regarding the filter, I misses the part where you can provide parameters > to filters, I can add it as a configuration option. Want to open an issue > for it? OK. I will open an issue for this. Also, couchdb supports user authentication. Some setups may require authentication for registering a stream. The url has to be '[http://user:pass@host](http://user:pass@host):port' Should I open another issue for this as well ? I guess this cannot be done with just the java.net.URL module. Mahendra [http://twitter.com/mahendra](http://twitter.com/mahendra) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 29, 2010, 10:32am UTC](https://discuss.elastic.co/t/queries-regarding-the-rivers-implementation/3383/8 "2010-09-29T10:32:11Z")
On Wed, Sep 29, 2010 at 12:27 PM, Mahendra M [mahendra.m@gmail.com](mailto:mahendra.m@gmail.com) wrote: > Hi Shay, > > On Wed, Sep 29, 2010 at 3:03 PM, Shay Banon wrote: > > > Yea, the \_river index acts as the registry for rivers and their > > metadata > > and state (providing persistency and state management for the rivers, not > > the data they index). For example, the coucdb one stores the last indexed > > seq in it. You can control the index name and type you index to. > > OK cool. So, I guess if one has to upload a mapping for an index > (river), it must be on the actual index and not the \_river index. Yea, you can precreate the index and put mappings and then create the river that indexes data from couchdb into it. > > Regarding the filter, I misses the part where you can provide > > parameters > > to filters, I can add it as a configuration option. Want to open an issue > > for it? > > OK. I will open an issue for this. > > Also, couchdb supports user authentication. Some setups may require > authentication for registering a stream. > > The url has to be '[http://user:pass@host](http://user:pass@host):port' > > Should I open another issue for this as well ? I guess this cannot be > done with just the java.net.URL module. Yea, open another issue. It can be done with the URL, not a problem. > Mahendra > [http://twitter.com/mahendra](http://twitter.com/mahendra) ---
**Author:** ![Mahendra\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mahendra_m/32/3128_2.png) [@Mahendra\_M](https://discuss.elastic.co/u/Mahendra_M)\ **Post date:** [September 29, 2010, 10:49am UTC](https://discuss.elastic.co/t/queries-regarding-the-rivers-implementation/3383/9 "2010-09-29T10:49:05Z")
Hi, I have opened the issues: > > > Couchdb filters support query parameters. It will be good to support this in the… Elasticsearch couchdb river plugin. > > Details of the couchdb filters are documented at > http://blog.couchone.com/post/468392274/whats-new-in-apache-couchdb-0-11-part-three-new. > > The parameters can be provider under: > > \`\`\` > { > "couchdb" : { > "filter" : "test", > "filter\_params" : { > "param1" : "value1", > "param2" : "value2" > } > } > } > \`\`\` > > > Some couchdb setups will require HTTP auth for listening on changes interface. T…he couchdb river plugin needs support for the same. > > Should add \`user\` and \`password\` under the \`couchdb\` section to support it. Regards, Mahendra On Wed, Sep 29, 2010 at 4:02 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > On Wed, Sep 29, 2010 at 12:27 PM, Mahendra M [mahendra.m@gmail.com](mailto:mahendra.m@gmail.com) wrote: > > > Hi Shay, > > > > On Wed, Sep 29, 2010 at 3:03 PM, Shay Banon wrote: > > > > > Yea, the \_river index acts as the registry for rivers and their > > > metadata > > > and state (providing persistency and state management for the rivers, > > > not > > > the data they index). For example, the coucdb one stores the last > > > indexed > > > seq in it. You can control the index name and type you index to. > > > > OK cool. So, I guess if one has to upload a mapping for an index > > (river), it must be on the actual index and not the \_river index. > > Yea, you can precreate the index and put mappings and then create the river > that indexes data from couchdb into it. > > > > Regarding the filter, I misses the part where you can provide > > > parameters > > > to filters, I can add it as a configuration option. Want to open an > > > issue > > > for it? > > > > OK. I will open an issue for this. > > > > Also, couchdb supports user authentication. Some setups may require > > authentication for registering a stream. > > > > The url has to be '[http://user:pass@host](http://user:pass@host):port' > > > > Should I open another issue for this as well ? I guess this cannot be > > done with just the java.net.URL module. > > Yea, open another issue. It can be done with the URL, not a problem. > > > Mahendra > > [http://twitter.com/mahendra](http://twitter.com/mahendra) -- Mahendra [http://twitter.com/mahendra](http://twitter.com/mahendra) ---
**Author:** ![Mahendra\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mahendra_m/32/3128_2.png) [@Mahendra\_M](https://discuss.elastic.co/u/Mahendra_M)\ **Post date:** [October 2, 2010, 3:52am UTC](https://discuss.elastic.co/t/queries-regarding-the-rivers-implementation/3383/10 "2010-10-02T03:52:55Z")
Hi Shay, On Wed, Sep 29, 2010 at 4:19 PM, Mahendra M wrote: > I have opened the issues: > [CouchDB River: Support couchdb filter query parameters · Issue #389 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/389) I tested this out. while forming the url params, the "=" was missing. I have sent a patch for it here. > > [Support http auth in couchdb river plugin · Issue #390 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/390) Was not able to test this one. Looks like there is an issue in couchdb w.r.t to auth for \_changes. Will clarify it and give it a try. Regards, Mahendra [http://twitter.com/mahendra](http://twitter.com/mahendra) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:18am UTC](https://discuss.elastic.co/t/queries-regarding-the-rivers-implementation/3383/11 "2017-07-06T04:18:33Z")