ElasticSearch indexing CouchDb views

Elastic Stack Elasticsearch
1

Hi all,
with the support of David Pilato I've downloaded a build of his extended
couchdb river plugin. The extended plugin allows you to configure
elasticsearch to index a view rather than _all_docs. Significantly this
capability offers a potential indexing performance increase by only
indexing selected parts of your document.

The plugin installed and ran without any problems under both 0.19 RC3 and
today's 0.19 Release.

You can download the plugin from
https://github.com/downloads/dadoonet/elasticsearch-river-couchdb/elasticseahttps://github.com/downloads/dadoonet/elasticsearch-river-couchdb/elasticsearch-river-couchdb-1.2.0-SNAPSHOT.zip
rch-river-couchdb-1.2.0-SNAPSHOT.ziphttps://github.com/downloads/dadoonet/elasticsearch-river-couchdb/elasticsearch-river-couchdb-1.2.0-SNAPSHOT.zip

Guidance in its use is at:
https://github.com/dadoonet/elasticsearch-river-couchdb/tree/dpi_views

Hopefully this new code will make it into the trunk fairly soon.

Well done David,
Ian

2

Thanks for the feedback Ian. It's good to know that other people test it before Shay merge the pull request in head.

Shay, if you agree with this pull request [1], could you merge it ?
Do you need more readme notes ?

David :wink:
@dadoonet

[1] Add support for views by dadoonet · Pull Request #2 · elastic/elasticsearch-river-couchdb · GitHub

Le 2 mars 2012 à 14:20, ian mayo ian@planetmayo.com a écrit :

Hi all,
with the support of David Pilato I've downloaded a build of his extended couchdb river plugin. The extended plugin allows you to configure elasticsearch to index a view rather than _all_docs. Significantly this capability offers a potential indexing performance increase by only indexing selected parts of your document.

The plugin installed and ran without any problems under both 0.19 RC3 and today's 0.19 Release.

You can download the plugin from https://github.com/downloads/dadoonet/elasticsearch-river-couchdb/elasticsea
rch-river-couchdb-1.2.0-SNAPSHOT.zip

Guidance in its use is at: https://github.com/dadoonet/elasticsearch-river-couchdb/tree/dpi_views

Hopefully this new code will make it into the trunk fairly soon.

Well done David,
Ian

3

On Fri, Mar 2, 2012 at 2:20 PM, ian mayo ian@planetmayo.com wrote:

Hi all,
with the support of David Pilato I've downloaded a build of his extended
couchdb river plugin. The extended plugin allows you to configure
elasticsearch to index a view rather than _all_docs. Significantly this
capability offers a potential indexing performance increase by only indexing
selected parts of your document.

The plugin installed and ran without any problems under both 0.19 RC3 and
today's 0.19 Release.

You can download the plugin
from https://github.com/downloads/dadoonet/elasticsearch-river-couchdb/elasticsea
rch-river-couchdb-1.2.0-SNAPSHOT.zip

Guidance in its use is
at: https://github.com/dadoonet/elasticsearch-river-couchdb/tree/dpi_views

Hopefully this new code will make it into the trunk fairly soon.

Well done David,
Ian

Does it means that it will ge all the results from the veiw each time
? Or do you get changes only ?

4

Only based on change API.

So when a doc change, we fetch the view using the doc id.

David :wink:
@dadoonet

Le 2 mars 2012 à 18:10, Benoit Chesneau bchesneau@gmail.com a écrit :

On Fri, Mar 2, 2012 at 2:20 PM, ian mayo ian@planetmayo.com wrote:

Hi all,
with the support of David Pilato I've downloaded a build of his extended
couchdb river plugin. The extended plugin allows you to configure
elasticsearch to index a view rather than _all_docs. Significantly this
capability offers a potential indexing performance increase by only indexing
selected parts of your document.

The plugin installed and ran without any problems under both 0.19 RC3 and
today's 0.19 Release.

You can download the plugin
from https://github.com/downloads/dadoonet/elasticsearch-river-couchdb/elasticsea
rch-river-couchdb-1.2.0-SNAPSHOT.zip

Guidance in its use is
at: https://github.com/dadoonet/elasticsearch-river-couchdb/tree/dpi_views

Hopefully this new code will make it into the trunk fairly soon.

Well done David,
Ian

Does it means that it will ge all the results from the veiw each time
? Or do you get changes only ?

5

On Fri, Mar 2, 2012 at 6:15 PM, David Pilato david@pilato.fr wrote:

Only based on change API.

So when a doc change, we fetch the view using the doc id.

hum ok. So maybe that :

http://lists.refuge.io/refuge/0122.html

will help ?

  • benoit
6

The main reason I haven't pulled this one in is because I am not too familiar with views and the right way to do it with couch, so the more input we can get from the community, the better :slight_smile:

On Friday, March 2, 2012 at 7:19 PM, Benoit Chesneau wrote:

On Fri, Mar 2, 2012 at 6:15 PM, David Pilato <david@pilato.fr (mailto:david@pilato.fr)> wrote:

Only based on change API.

So when a doc change, we fetch the view using the doc id.

hum ok. So maybe that :

http://lists.refuge.io/refuge/0122.html

will help ?

  • benoit
7

Sure.

Note that if users don't use this new feature (let's say experimental), there is no cost.

So, I think it's safe to pull it in.

BTW, I will look at what Benoist suggest to see if the code could be more efficient. But, it can be a new issue (change request) for the couchDb plugin.

David :wink:
@dadoonet

Le 3 mars 2012 à 23:08, Shay Banon kimchy@gmail.com a écrit :

The main reason I haven't pulled this one in is because I am not too familiar with views and the right way to do it with couch, so the more input we can get from the community, the better :slight_smile:
On Friday, March 2, 2012 at 7:19 PM, Benoit Chesneau wrote:

On Fri, Mar 2, 2012 at 6:15 PM, David Pilato david@pilato.fr wrote:

Only based on change API.

So when a doc change, we fetch the view using the doc id.

hum ok. So maybe that :

http://lists.refuge.io/refuge/0122.html

will help ?

  • benoit
8

Hi there David,
has the views code been integrated into the trunk yet?

I saw in your comment [1] that you'd merged with 0.19, but the CouchDb
river readme still doesn't include reference to support for views.

cheers,
Ian

[1]. https://github.com/elasticsearch/elasticsearch-river-couchdb/pull/2#commits-pushed-a35917b