Logstash config to delete records in elastic?

Elastic Stack Logstash
1

Hi,

I'm using Logstash 5.1.1 and new to the ELK stack. I have a logstash config file setup that does an upsert of records from my Microsoft SQL Server database every 5 minutes into an index in Elastic. When a record or records(s) are deleted from SQL Server, I want to somehow delete the records in the elastic index to keep things in sync.

Can someone kindly share a config that will do that or point me maybe somewhere that illustrates how this is done? I'm happy to learn, just need a bit of help. My end goal is to ensure that if a record is deleted in my source database, it also gets deleted in elastic.

I can provide my current config if it helps.

2

You might be able to pull this off with an elasticsearch input and a jdbc_streaming filter. The idea would be to fetch all documents from ES and look them up in the database. If you don't get a match you need to issue a deletion request. This is probably possible but the configuration might become non-trivial and most likely won't be very efficient.

3

Thanks for the reply, it was helpful and made me think a bit more about the requirement I was trying to solve for. I think I may be better of when a user needs to delete something I delete it first from elasticsearch index and then from my database versus the other way around. Or perhaps just do a soft delete (e.g. active flag or something) in my database and do an update on the index.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.