Indexing different types from a CouchDB database

Mark_Huang_2 · August 13, 2012, 9:38am

Hi,

Reading http://www.elasticsearch.org/blog/2010/02/12/yourdatayoursearch.html,
I realized my problem is exactly the same as the article except for one
huge difference: Instead of indexing one book or cd at a time, I already
have a bunch of data in my couchdb database. I'm adding elastic search
capabilities to my project now and I'm having a ton of problems finding out
how to index my couchDB database by type.

The documentation says that one way to index ALL documents is like the
following:

curl -XPUT 'localhost:9200/_river/amazon/_meta' -d '{
"type" : "couchdb",
"couchdb" : {
"host" : "localhost",
"port" : 5984,
"db" : "amazon",
"filter" : null
},
"index" : {
"index" : "amazon",
"type" : "book",
"bulk_size" : "100",
"bulk_timeout" : "10ms"
}
}'

Here I specified the index being amazon and book being the type. But this still indexes ALL the documents in CouchDB! How do I make it ONLY index the book type documents? Is this what the filter is for? Or can this be accomplished with script field: ctx.type = "book" <-- Tried this but this didn't work either.

Thanks in advance!

--

dadoonet · August 13, 2012, 10:01am

I think that you can use ctx.ignore=true
To ignore some documents.

Read here GitHub - elastic/elasticsearch-river-couchdb: CouchDB River Plugin for elasticsearch (STOPPED)

BTW, I wrote a support for CouchDb views but it's not merged.

github.com/elastic/elasticsearch-river-couchdb

Add support for views

elastic:master ← dadoonet:dpi_views

opened 09:06PM - 05 Dec 11 UTC

dadoonet

+410 -55

Moving from https://github.com/elasticsearch/elasticsearch/issues/1512 Real imp…lementation of pull request https://github.com/elasticsearch/elasticsearch/issues/1258, as there was a problem with GitHub. In CouchDB, you can retrieve docs by `GET`, `_changes` API and views. CouchDB river uses `_changes` API to get documents. I would like to be able to get documents that changed (getting ID with the _changes API) using a view with parameter `key="DOCID"`. As views return a collection of results (aka rows), we will index in ES each row with an id like DOCID_seq where seq is the sequence number of each row. If you get back 3 rows for one single change for document with ID=1234, the river will index 3 documents : - 1234_1 - 1234_2 - 1234_3 To use it, you have to define a view in couchDB. For instance, `_design/vues/_view/test_dpi` with ``` javascript function(doc) { listArt=doc.document.articles; for(var i=0; i<listArt.length;i++) { var artJson = {}; artJson = { 'docid' : doc._id, 'num' : listArt[i].numeroArticle }; artJson = JSON.stringify( artJson ); emit(doc._id, eval('('+artJson+')') ); }; } ``` You can use it in your couchDb river as follow : ``` javascript { "type":"couchdb", "couchdb": { "host":"localhost", "port":"5984", "db":"dau_test", "view":"vues/_view/test_dpi", "viewIgnoreRemove":false } } ``` New options : - `view` : if not null, couchDB river will not fetch content from `_changes` API but only IDs and then will use the view to retrieve rows using the ID as a key. By default : null - `viewIgnoreRemove` : ask the river to ignore removal of rows if there is less rows after a document update. By default : false so non existing rows will be removed from elastic search. For example, with the 3 rows described earlier, if you push a new version of the document 1234 in couchDB with only 2 docs, If `viewIgnoreRemove` is false (default), then - 1234_1 will be updated - 1234_2 will be updated - 1234_3 will be removed If `viewIgnoreRemove` is true, then - 1234_1 will be updated - 1234_2 will be updated - 1234_3 will not be updated BTW, I will push an update to the pull request when the `ids_prefix` filter will be available to make code more efficient. (See issue https://github.com/elasticsearch/elasticsearch/issues/1259). Thanks David

David

--

Le 13 août 2012 à 11:38, Mark Huang zhenghao12@gmail.com a écrit :

Hi,

Reading Elasticsearch Platform — Find real-time answers at scale | Elastic, I realized my problem is exactly the same as the article except for one huge difference: Instead of indexing one book or cd at a time, I already have a bunch of data in my couchdb database. I'm adding Elasticsearch capabilities to my project now and I'm having a ton of problems finding out how to index my couchDB database by type.

The documentation says that one way to index ALL documents is like the following:

curl -XPUT 'localhost:9200/_river/amazon/_meta' -d '{
"type" : "couchdb",
"couchdb" : {
"host" : "localhost",
"port" : 5984,
"db" : "amazon",
"filter" : null
},
"index" : {
"index" : "amazon",
"type" : "book",
"bulk_size" : "100",
"bulk_timeout" : "10ms"
}
}'
Here I specified the index being amazon and book being the type. But this still indexes ALL the documents in CouchDB! How do I make it ONLY index the book type documents? Is this what the filter is for? Or can this be accomplished with script field: ctx.type = "book" <-- Tried this but this didn't work either.
Thanks in advance!

--

Mark_Huang_2 · August 13, 2012, 10:16am

Are you saying it should be like this:

curl -XPUT 'localhost:9200/_river/amazon/_meta' -d '{
"type" : "couchdb",
"couchdb" : {
"host" : "localhost",
"port" : 5984,
"db" : "amazon",
"filter" : null,
"script": "ctx.ignore = true; ctx.doc.name = ctx.doc.content.name
;"
},
"index" : {
"index" : "amazon",
"type" : "book",
"bulk_size" : "100",
"bulk_timeout" : "10ms"
}
}'

The documentation says : Also, if ctx.ignore is set to true, the change
seq will be ignore and not applied.
Wouldn't this mean that the book document will not be applied? Can I add
if else statements? if (ctx.type !== 'book') ctx.ignore = True; ?

On Mon, Aug 13, 2012 at 5:01 AM, David Pilato david@pilato.fr wrote:

I think that you can use ctx*.ignore=true *
To ignore some documents.

Read here GitHub - elastic/elasticsearch-river-couchdb: CouchDB River Plugin for elasticsearch (STOPPED)

BTW, I wrote a support for CouchDb views but it's not merged.
Add support for views by dadoonet · Pull Request #2 · elastic/elasticsearch-river-couchdb · GitHub

David

--

Le 13 août 2012 à 11:38, Mark Huang zhenghao12@gmail.com a écrit :

Hi,

Reading
Elasticsearch Platform — Find real-time answers at scale | Elastic, I
realized my problem is exactly the same as the article except for one huge
difference: Instead of indexing one book or cd at a time, I already have a
bunch of data in my couchdb database. I'm adding Elasticsearch
capabilities to my project now and I'm having a ton of problems finding out
how to index my couchDB database by type.

The documentation says that one way to index ALL documents is like the
following:

curl -XPUT 'localhost:9200/_river/amazon/_meta' -d '{
"type" : "couchdb",
"couchdb" : {
"host" : "localhost",
"port" : 5984,
"db" : "amazon",
"filter" : null
},
"index" : {
"index" : "amazon",
"type" : "book",
"bulk_size" : "100",
"bulk_timeout" : "10ms"
}
}'

Here I specified the index being amazon and book being the type. But this still indexes ALL the documents in CouchDB! How do I make it ONLY index the book type documents? Is this what the filter is for? Or can this be accomplished with script field: ctx.type = "book" <-- Tried this but this didn't work either.

Thanks in advance!

--

--

--

Dan_Everton · August 13, 2012, 11:44pm

I'm assuming you have a CouchDB database that has multiple types of
documents in it and you only want to index one of those types? You can do
that in two ways:

A filter function on the CouchDB
A filter function on the Elasticsearch side.

The first way requires you to add a filter function to a design document in
the CouchDB database. So in your design document you'll have something like:

{
"_id" : "_design/books",
"filters" : {
"only" : "function(doc, req) { return doc.type == 'book' }"
}
}

And then your river definition looks like this:

{
"type" : "couchdb",
"couchdb" : {
"host" : "localhost",
"port" : 5984,
"db" : "amazon",
"filter" : "books/only"
},
"index" : {
"index" : "amazon",
"type" : "book"
}
}

The second way to do it is to use a script on the Elasticsearch side. This
is essentially the same as the previous way, but puts the script execution
burden on ES not CouchDB. So your river definition in this case looks like:

{
"type" : "couchdb",
"couchdb" : {
"host" : "localhost",
"port" : 5984,
"db" : "amazon",
"script" : "ctx.ignore = ctx.doc.type != 'book'"
},
"index" : {
"index" : "amazon",
"type" : "book"
}
}

Note that this river definition requires the Javascript plugin to be added
to your Elasticsearch install.

Cheers,
Dan

--

Mark_Huang_2 · August 14, 2012, 3:36am

Thanks a million Dan! You have saved me from complete despair to full of
hope!

Elasticsearch CouchDB river is great, but (no offense), I feel it lacks
some proper usage examples. Maybe it's just me, but I doubt any n00b can
understand what to do with ctx.ignore just by reading the documentation.

One FINAL thing:

Is it possible to save the javascript "script" field value into a file and
then reference it from the river definition?

On Monday, 13 August 2012 18:44:51 UTC-5, Dan Everton wrote:

I'm assuming you have a CouchDB database that has multiple types of
documents in it and you only want to index one of those types? You can do
that in two ways:

A filter function on the CouchDB

A filter function on the Elasticsearch side.

The first way requires you to add a filter function to a design document
in the CouchDB database. So in your design document you'll have something
like:

{
"_id" : "_design/books",
"filters" : {
"only" : "function(doc, req) { return doc.type == 'book' }"
}
}

And then your river definition looks like this:

{
"type" : "couchdb",
"couchdb" : {
"host" : "localhost",
"port" : 5984,
"db" : "amazon",
"filter" : "books/only"
},
"index" : {
"index" : "amazon",
"type" : "book"
}
}

The second way to do it is to use a script on the Elasticsearch side. This
is essentially the same as the previous way, but puts the script execution
burden on ES not CouchDB. So your river definition in this case looks like:

{
"type" : "couchdb",
"couchdb" : {
"host" : "localhost",
"port" : 5984,
"db" : "amazon",
"script" : "ctx.ignore = ctx.doc.type != 'book'"
},
"index" : {
"index" : "amazon",
"type" : "book"
}
}

Note that this river definition requires the Javascript plugin to be added
to your Elasticsearch install.

Cheers,
Dan

--

Mark_Huang_2 · August 14, 2012, 4:37am

What if after indexing the book type, I now want to index another type
called "cd". Do I just create a second _river definition? I tried
creating a second _river definition to index cd but nothing seemed to
happen. Help?

On Monday, 13 August 2012 18:44:51 UTC-5, Dan Everton wrote:

I'm assuming you have a CouchDB database that has multiple types of
documents in it and you only want to index one of those types? You can do
that in two ways:

A filter function on the CouchDB

A filter function on the Elasticsearch side.

The first way requires you to add a filter function to a design document
in the CouchDB database. So in your design document you'll have something
like:

{
"_id" : "_design/books",
"filters" : {
"only" : "function(doc, req) { return doc.type == 'book' }"
}
}

And then your river definition looks like this:

{
"type" : "couchdb",
"couchdb" : {
"host" : "localhost",
"port" : 5984,
"db" : "amazon",
"filter" : "books/only"
},
"index" : {
"index" : "amazon",
"type" : "book"
}
}

The second way to do it is to use a script on the Elasticsearch side. This
is essentially the same as the previous way, but puts the script execution
burden on ES not CouchDB. So your river definition in this case looks like:

{
"type" : "couchdb",
"couchdb" : {
"host" : "localhost",
"port" : 5984,
"db" : "amazon",
"script" : "ctx.ignore = ctx.doc.type != 'book'"
},
"index" : {
"index" : "amazon",
"type" : "book"
}
}

Note that this river definition requires the Javascript plugin to be added
to your Elasticsearch install.

Cheers,
Dan

--

Dan_Everton · August 14, 2012, 5:27am

On Tuesday, August 14, 2012 2:37:58 PM UTC+10, Mark Huang wrote:

What if after indexing the book type, I now want to index another type
called "cd". Do I just create a second _river definition? I tried
creating a second _river definition to index cd but nothing seemed to
happen. Help?

If everything's in the same CouchDB database you can just use a single
river definition and use a script on the Elasticsearch side to set the
type. So say you have a "type" field in your documents in CouchDB that is
also the same as the type you want to use in Elasticsearch. Your river
definition would look like this:

{
"type" : "couchdb",
"couchdb" : {
"host" : "localhost",
"port" : 5984,
"db" : "amazon",
"script" : "ctx._type = ctx.doc.type"
},
"index" : {
"index" : "amazon"
}
}

Setting ctx._type tells Elasticsearch what type in the index to use. So if
doc.type for a CouchDB changeset is "book" then Elasticsearch will index it
as the "book" type. Obviously you can make the script a bit more
sophisticated if your types don't map one to one. I'm not aware of any way
to externalise that script unfortunately.

Cheers,
Dan

--

Mark_Huang_2 · August 14, 2012, 5:39am

Damn....I feel so stupid for not realizing that this could be done. I read
about the Elasticsearch _type.....but wasn't sure what to make of it.

Thank you once again

On Tuesday, 14 August 2012 00:27:13 UTC-5, Dan Everton wrote:

On Tuesday, August 14, 2012 2:37:58 PM UTC+10, Mark Huang wrote:

What if after indexing the book type, I now want to index another type
called "cd". Do I just create a second _river definition? I tried
creating a second _river definition to index cd but nothing seemed to
happen. Help?

If everything's in the same CouchDB database you can just use a single
river definition and use a script on the Elasticsearch side to set the
type. So say you have a "type" field in your documents in CouchDB that is
also the same as the type you want to use in Elasticsearch. Your river
definition would look like this:

{
"type" : "couchdb",
"couchdb" : {
"host" : "localhost",
"port" : 5984,
"db" : "amazon",
"script" : "ctx._type = ctx.doc.type"
},
"index" : {
"index" : "amazon"
}
}

Setting ctx._type tells Elasticsearch what type in the index to use. So if
doc.type for a CouchDB changeset is "book" then Elasticsearch will index it
as the "book" type. Obviously you can make the script a bit more
sophisticated if your types don't map one to one. I'm not aware of any way
to externalise that script unfortunately.

Cheers,
Dan

--

Mark_Huang_2 · August 14, 2012, 7:43am

So I had this requirement which David Pilato solved for me:
https://groups.google.com/d/msg/elasticsearch/KRPldny3wUk/nrfJtkOsv-oJ

As you can see, a book may have a different output fields when compared
with a cd. And hence I wanted a way to index different types (matching my
couchdb type with Elasticsearch type) separately. If I were to use:

script: ctx._type = ctx.doc.type, I wouldn't be able to set the different
output fields per document type. Someone else told me that I could use
partial fields to accomplish this, is this right?