Want to delete documents from elasticsearch which are deleted from database

[milankamboya (Milan)]

Hi All,

I want to delete records from Elasticsearch for below use-case. What are the possible resolutions?

Fetching data from database through logstash and adding into Elasticsearch.
document_id is combination of key + some additional information.
logstash is running every few minutes to fetch latest created/modified data based timestamp.
In my scenario, we have history table for every table. So, when record has been deleted from main table, it will be added into history table with DELETE action.
I have 4-5 indexes for different purposes.
Now, I want to update/remove this related record from Elasticsearch indexes.
What are the best resolutions?

My thoughts

1. are there any facility using logstash or any other plugin through which I can fetch those deleted records and delete from Elasticsearch?
2. is it good to implement cron job using spring-boot to deleted records from Elasticsearch?

[mverbeek (Marc)]

You can delete specified records from Elastic with the 'Delete by query API'.

You could put this into a cron job but it would not be the most clean solution.

[milankamboya (Milan)]

Is this direct? or using logstash?
Any other approach?

[dadoonet (David Pilato)]

Not exactly the answer but I shared most of my thoughts regarding database sync there: http://david.pilato.fr/blog/2015/05/09/advanced-search-for-your-legacy-application/

Basically, I'd recommend modifying the application layer if possible and send data to elasticsearch in the same "transaction" as you are sending your data to the database.