Hello.
I am continuously listening to data from a receiver. It continuously sends the data to a port in json format. I need to parse and save the data I receive from the receiver very quickly. When I write a server for this with Mongodb, I can log around 200 logs per second and there is no loss. But when I write for elastic, I lose my logs and it saves 5 logs per second. For me, it is important that I can make a quick query after saving the data, so I preferred elastic. Is there a different method of saving between mongodb and elasticsearch? For Elastic, I think a log is made with HTTP requests. Is there a problem because of this? Or is there a problem with my code optimization?
In order to efficiently ingest large amounts of data into Elasticsearch it is recommended to use the bulk API. This means that you typically batch up multiple requests, which increases the amount of time until the document is searchable. Elasticsearch is also not designed for real-time search and only makes documents searchable once a second (default value) once the data has been indexed. This is due to the process of making the documents searchable being quite expensive. It is possible to lower this, but that typically comes at the price of performance.
If your requirement is to quickly index documents one by one and immediately have them searchable Elasticsearch is likely not the best option.
1 Like
I understand, thank you. I have one more question. How is elasticsearch compatible with another database system? I realized that you can save data to a nosql (like mongodb) database on the internet and search with elasticsearch. How does synchronization work here or can work be done efficiently?
I am not familiar with integrations between MongoDB and Elasticsearch, but know that there is a connector available for Elastic Enterprise Search. I have however not used this. Note that any such integration will be subject to the constraints I mentioned earlier and near real-time in nature, not immidiate.
1 Like
thank you for everything
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.