How to keep Database and Elasticsearch consistent using Logstash


(Patrick) #1

I have a Relational database, I have used Logstash to migrate my Relational Postgres data to elasticsearch.

Now how can I keep my Database and Elasticsearch Server in sync?


(David Pilato) #2

I shared most of my thoughts there: http://david.pilato.fr/blog/2015/05/09/advanced-search-for-your-legacy-application/

Basically, I'd recommend modifying the application layer if possible and send data to elasticsearch in the same "transaction" as you are sending your data to the database.


(Patrick) #3

@dadoonet Is there any other way of doing it ?


(David Pilato) #4

Did you read my blog? The start of it at least?


(Patrick) #5

@dadoonet Yes I had gone through it earlier. The blog is very helpful. It is great help

I was able to find the solution I used 'record_last_run' to record the last run the logstash and used the scheduler to check for changes in the Database


(David Pilato) #6

Awesome.