Hello all,
We are running a project using Nodejs(loopback for api creation) and MongoDB as out database. We have around 1.4 - 2 million records. Now the real concern is the searching, it has become very slow. Most of the time a find query to fetch the 50 latest added records would give us connection timeout error. We have indexed our DB completely so to try and make searching fast, but it doesn't seem to affect much.
At this point, we want to introduce elasticsearch so to execute queries faster, but not able tot find best solution or architecture to use with MongoDB. Please suggest some ways in which we can accommodate elasticsearch in our project along with mongodb.
Open to any ideas.
Thanks.
My advice: Update you application and send your documents to elasticsearch at the same time you are sending them to MongoDB.
I would like to add the following to the advice above.
Let's say you prefer to store data in MongoDB and would like to search in ES. You insert a document into MongoDB first. MongoDB will generate an _id for the newly inserted document. Take this _id and index the document in ES with this _id and tell ES to index but don't store the content (to save some space) so when you search, you search through ES, and to view the document, use the _id value returned from ES to retrieve the document in MongoDB.
Additionally, later one, if you update a document in MongoDB and would like to update ES, index the updated document in MongoDB using the same _id value in ES, ES will replace the old content with the new content, this will prevent duplication in ES. By doing this, you have to insert documents into MongoDB first, then index them with ES and the associated _id generated from MongoDB.
If you need to do things in parallel, try to be creative with the _id to avoid having duplicated documents in ES.
If space in ES is not your real concern, you can let ES keep the document when indexing it so you can retrieve the document directly from ES, instead of making a trip to MongoDB. If this is what you prefer, then think why you need MongoDB and adjust your requirements accordingly.
Thanks, @dadoonet and @thn for your replies.
I can think of two approaches right now, please help me choose best.
I can maintain two DB,
- Mongo for all data. (actual DB)
- Elastic for dirty searching and giving it only data, which needs to be searched (not only id but replicate the whole document so to enable search on basis of other fields also). Now to update elastic, I could use Logstash to register changes(insert and update) from mongo to elastic with this input plugin and this output plugin. Is this approach recommended?
or
I can maintain only one DB,
Elastic for everything, no need for logstash, but is elastic a good choice to save sensitive data, as users and payments info?
Thanks again.
Well... Elasticsearch is not a database. Read this page before taking any decision so you are aware of what can happen...
Thanks for the information. Now, we will be making an informed decision. 
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.