I am creating an river plugin that synchronizes ES and a postgresql
database. I work with an timestamp to check if my document in the database
has changed, if it has changed it is re-indexed in ES if it has not
changed, my plugin skips the document. My timestamp is a column in my
database and it is also indexed with my document. My problem is, that I
don't know how I can synchronize documents that are deleted in the
database? So my id of my document is no longer available in my database,
but how can I tell ES that it has to delete this document, if I do no
longer have the id to delete it. Does anyone has an idea how I could solve
I connect to my database, select all documents out of my database and
iterate through my result set. Then I search for each id in ES and check
timestamp from database with timestamp of indexed document in ES. But if my
document is no longer available in my database, I don't know how to inform
ES that a document has been deleted in my database.
I have looked at the jdbc river plugin, and there is used a version for
each move of the river and if the document is deleted in the database, the
version is lower than in ES, so all documents are deleted. But in my case I
only update documents if they have changed, so only those documents that
are updated are getting a higher version, and those documents that are not
changed in the database, have the same version as before. So if I would
also delete documents like it is done in the jdbc river plugin, I also
would delete documents that are still available in my database, but not
updated because they have not changed in the database. I don't think that
this solution is appropriate for my problem.
Does anyone knows if other river plugins are available, that could help me
with my problem? Or does anyone know a solution, to synchronize deleted
documents in ES?
Thanks for any help.
Hopefully I described my problem in a understandable way.
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to email@example.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ce722fe4-6d17-4b23-9d4c-e4f5f49ed349%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.