I am researching on how to sync relation database to elastic search using EsRDD. I found the EsRDD doesn't support delete operation from previous conversation in this forum. I encounter a problem, if a row is deleted in relation db, how could I do it in ESRDD? I figure out some resolutions,
- use the rest API of elasticsearch for deleting, other operations are done by EsRDD (add update)
- add an additional key value in every index, for example, "really_exist_flag": "true/false"
When the user queries data, he should filter "really_exist_flag" which is "false" firstly.
- update all key-value in this index to empty value, so these values will not be found
I don't know the efficiency of the elasticsearch on deleting the index. Let us assume that the relation db has many delete operations. Could you give me some advice for it? OR is there any other method on how to sync relation db to elasticsearch ?