Syncing Mongodb with Elasticsearch

Elastic Stack Elasticsearch
1

Hi, I wish to transfer data from Elasticsearch into a Mongodb instance. The data in Elasticsearch is nearly 120 GBs and is continuoly updating with nearly 4GB per day. How do I do it?
What I have tried:

  • Logstash is very slow and also the problem is that on stopping logstash it restarts from the beginning again (https://github.com/logstash-plugins/logstash-input-elasticsearch/issues/93)
  • Scroll Api is good for migrating large data, but what about real time data? The documentation doesn't recommend using it for real time data.
  • Pagination is very slow, and it might work but in lot of discussions it is not recommended for documents in the size of millions. I have tried in less number of documents and it works fine but how do I make sure it will work on a scale too?

Any suggestions are welcome.

2

Hey,

googling around shows quite a few solutions. I don't know if there is the one and best solution, but I have heard some good things of monstache - see https://rwynn.github.io/monstache-site/ . Never used it myself though!

--Alex

3

Hi Alex,
Thanks for your answer but I think you misunderstood the question. I wish to get data from Elastisearch into MongoDB, not the other way around. Monstache does the opposite of my requirement.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.