I'm using Logstash 5.1.1 and new to the ELK stack. I have a logstash config file setup that does an upsert of records from my Microsoft SQL Server database every 5 minutes into an index in Elastic. When a record or records(s) are deleted from SQL Server, I want to somehow delete the records in the elastic index to keep things in sync.
Can someone kindly share a config that will do that or point me maybe somewhere that illustrates how this is done? I'm happy to learn, just need a bit of help. My end goal is to ensure that if a record is deleted in my source database, it also gets deleted in elastic.
I can provide my current config if it helps.
You might be able to pull this off with an elasticsearch input and a jdbc_streaming filter. The idea would be to fetch all documents from ES and look them up in the database. If you don't get a match you need to issue a deletion request. This is probably possible but the configuration might become non-trivial and most likely won't be very efficient.
Thanks for the reply, it was helpful and made me think a bit more about the requirement I was trying to solve for. I think I may be better of when a user needs to delete something I delete it first from elasticsearch index and then from my database versus the other way around. Or perhaps just do a soft delete (e.g. active flag or something) in my database and do an update on the index.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.