Need a design/approach to build a custom sync engine between elasticsearch and RestAPIs

Elastic Stack Elasticsearch
1

Hi there,

My application is interacting with elasticsearch instance for all data retrival and searchability. Elasticsearch needs to feed data from RestAPI (Bulk data around 2-4 gigs). For that I need to write my rest Client to feed elasticsearch with the rest api. Which is perfectly alright.

The Sync engine will dump all the data from REst APIs to Elasticsearch (one time activity) and than we need keep updating the ES instance with the newly added/updated/deleted fields at mongoDB. Since there is no direct channel to intract with mongoDB, its difficult to fetch the updated information from mongo.

Also there is no field called "Last updated" in mongoDB.I need to write a custom transporter utility which will run on a fixed time, which will fetch the data from rest apis, compare it with elasticsearch and update the modified content only.

Can we use any of the existing plugin like http-poller ? Please help !!!!

Thanks

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.