JDBC input MS SQL

Elastic Stack Logstash
1

Hi, everyone!
I try to move my data from MS SQL DB to elastic. I found JDBC plugin and it's cool =)

But I need to store in elastic only actual data that exist at this moment in SQL DB.

How can I do this correct? I mean how I can delete old recors in elastic that doesn exist in SQL anymore

2

This is a pretty classic data synchronisation problem, and strategies here aren't particularly limited to Logstash/Elasticsearch :slight_smile:

  • Do you have access to a transaction log, where you can detect a row being deleted from the SQL DB? If so, that will be your best bet.
  • Another option would be to tag each document with a "last seen" timestamp, and periodically go back and "collect" the documents that haven't been seen recently (this could be done in a separate Logstash pipeline with elasticsearch input and elasticsearch output).
  • Depending on the size of your dataset and how frequently you synchronize, you may be best off simply reading the data into a new index each time you import.
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.