Data base Synchronize with elastic search in real time

Elastic Stack Elasticsearch
1

Hi,

I am new to elastic search, the data being inserted into the database I wanna load the data into elastic search. when new record has being inserted into database elastic search has to Synchronize with DB automatically. I was using DB2 what wil be the best practice.

2

I shared my thoughts there: http://david.pilato.fr/blog/2015/05/09/advanced-search-for-your-legacy-application/

3

Thank you, David.

I need to synch elastic search because the data inserting into database is not form my application. It's from the Sterling Integrator so i need to sysnc it so could you suggest me the the best aproach.

4

Then you need to use/build something which reads the database, transform to JSON and write to elasticsearch.

  • An ETL like Logstash with the jdbc-input-plugin
  • Your own code

Then you can run that every 5 minutes for instance but as I explained, you have to deal with many things:

  • It's not real-time
  • You need a way to select data that changed since the last run. Probably it relies on a date field in your DB entities
  • You need to deal with Deletes. You need for that a specific technical table in your DB probably.

Or you can choose to reindex the full database everytime. Be careful with the time it will take to read entities from your DB. May be you can do such a thing only once per day.

May be you can play with DB triggers though to "wake up" your code.

2 Likes
5

Thank you very much, David,

I in some other blog I have seen using logstash I can load the data into the elasticsearch all at once. but I want to load the data when a new record has been inserted only the new record has to load into ES. It's a kind of trigger or something. Do I Have an option as I needed?

6

Anything which looks like a trigger basically depends from your database vendor.

For example, CouchDB has this super nice changes API which streams changes over the wire. I don't know if something similar can exist in your DB.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.