CouchDB-River deleted document

Elastic Stack Elasticsearch
1

Hello,

I think I am missing something obvious but when I delete a CouchDb document
the river doesn't seem to be picking up on the change.

This is how I create the river:

curl -XPUT 'localhost:9200/_river/products/_meta' -d '{
"type" : "couchdb",
"couchdb" : {
"host" : "localhost",
"port" : 5984,
"db" : "fluf",
"filter": "general/products"
},
"index" : {
"index" : "products",
"type" : "product",
"bulk_size" : "100",
"bulk_timeout" : "10ms"
}
}'

All my couchDb documents have a type field and the general/products filter
will filter on type === 'product'. So is there something I am missing here?

Friendly Greetings,

Geert

2

When CouchDB deletes a document, the change event does not include the
document. So if your filter function is something like:

return doc.type == 'product'

then it's filtering out the delete events. There's nothing that
Elasticsearch can do in this case unfortunately. You just have to
design around it by not filtering delete events.

See https://github.com/elasticsearch/elasticsearch-river-couchdb/issues/7
for more discussion about this corner case.

Cheers,
Dan

3

Hello,

does it actually work if I remove the filter or do I have to do something
else?

I read that issue a few times before but I guess I am not fully
understanding it. If you delete something the change feed will show a
delete message with the document id right? The document id is unique inside
a couchdb database so couldn't you use that to delete it from the index or
is it a problem if one elasticsearch instance is feeding of different
couchDB databases so you can't be sure the id is unique?

4

If you delete a document in couchDb, changes API will send the delete in stream and ES river will delete the document in ES as well.

HTH
David :wink:
@dadoonet

Le 13 févr. 2012 à 12:36, Geert Pasteels geert.pasteels@gmail.com a écrit :

Hello,

does it actually work if I remove the filter or do I have to do something else?

I read that issue a few times before but I guess I am not fully understanding it. If you delete something the change feed will show a delete message with the document id right? The document id is unique inside a couchdb database so couldn't you use that to delete it from the index or is it a problem if one elasticsearch instance is feeding of different couchDB databases so you can't be sure the id is unique?

5

I just realized why it doesn't work with a filter I think. Cause the delete
message doesn't show up in the changes feed if you filter it?

6

CouchDB is sending the delete message but your filter function is
preventing it from going to Elasticsearch.

I assume your filter function looks like this"

return doc.type == 'product'

Well that will filter out deleted document events because the only
fields a deleted document as are _id, _rev, and _deleted. By saying it
must also have a type field, your filter is rejecting the delete
event.

If you want the delete events to pass through your filter you need to
do something like

return doc._deleted || doc.type == 'product'
7

That's a great solution Dan, thanks a bunch!