Elasticsearch and Mongo DB real time sync

Elastic Stack Elasticsearch
1

I have an use case where Mongo DB is my primary data store and search API needs to connect to elasticsearch for getting the list of inventory.

I need to have real time sync between elasticsearch and mongo db for two scenarios.

Scenario:1 : New items gets added to existing inventory (This is relatively easy to solve. I will have a management api that will do concurrent update to Mongo and elasticsearch, so records will be consistent across both)

Scenario:2: Items are limited in an inventory, example 10 items in an inventory and with each purchase count decreases so it rises the condition of realtime update of mongodb and elasticsearch. Say there is a sync difference of 1 min, which means that even if all the items are sold out for some specific inventory, that record be picked up by the search query until next sync, and users will get stale deals in that 1 mins sync delay.

2

There isn't really an automated way to do this that I'm aware of. You'll have to replicate each action (index, update, delete, etc) to both Mongo and Elastic in realtime. E.g. when a user purchases something and the inventory decreases, you'll need to immediately replicate that update action to Elasticsearch.

1 Like
3

Understand. But Mongodb do not provide native support for triggers. Is there any other way?

4

I have no idea to be honest, I know very little about Mongo. :yum:

I know some DBs can stream their actions live ("Changes Feed" in CouchDB, "Oracle Streams" in Oracle, etc)...unsure if Mongo has the equivalent.

If it doesn't, I think you'll just have to bake this logic into your application. Every time an operation happens, replicate the action to both systems

1 Like
5

Thanks Zachary

6

@Mark_Harwood just pointed out to me that Mongo has an oplog which some people tail for this sort of thing. E.g. here is a node package which monitors the oplog and fires an event for each different type of operation: https://github.com/cayasso/mongo-oplog

A quick google shows people using the oplog for a variety of replication tasks, so you could probably use that for ES too. Goodluck!