How to sync only updated data from mysql table with logstash is it possible?

Elastic Stack Logstash
1

how to sync only updated data from mysql table with logstash is it possible with logstash configuration

2

Welcome!

I think that you'd need a technical field within your data which is the update date of the row. Then you could hopefully select whatever is more than your last run.

That means:

  • Read from the database (SELECT * from TABLE)
  • Convert each record to a JSON Document
  • Send the json document to elasticsearch, preferably using the _bulk API.

Logstash can help for that. But I'd recommend modifying the application layer if possible and send data to elasticsearch in the same "transaction" as you are sending your data to the database.

I shared most of my thoughts there: https://david.pilato.fr/blog/2015-05-09-advanced-search-for-your-legacy-application/

Have also a look at this "live coding" recording.