MySQL Syncing with Logstash to Elasticsearch

Elastic Stack Logstash
1

Hello,

For the last few hours i have been trying to implement synchronisation between MySQL database and Elasticsearch using Logstash and the JDBC plugin.
Each row in each table has a updated_at column (timestamp, refreshed when something changes in the row), created_at column (timestamp, only set when the row is created) and a deleted column (boolean, set to true when the backend - play framework - deletes something, no hard deletes).
What i want to do is query all rows from all tables that have been updated / created since the last check, and add those that have been just created (created_at > last check), update existing documents (updated_at > last check) and delete those that have their deleted flag set to true and updated_at > last check.
Im having a real trouble figuring out how to do this and any help, links or pointers would be appreciated :slight_smile:

PS: If i understand correctly each table should have its own index, right? All examples on the internet only check one table :confused:

2

Is the updated_at column set for newly created documents, i.e. do you actually need to care about created_at?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.