What is the best way of getting MongoDB data into Elasticsearch?

Elastic Stack Elasticsearch
1

Hi All,

Now that rivers are being deprecated, I need a more future proof easy way of replicating MongoDB data to our Elasticsearch cluster, what would be the best solution for this?

We don't need to transform the data before storing it in Elasticsearch, for now we just want to index everything. I definitely want to avoid baking stuff into our code and re-inventing wheels.

Thanks
Warren.

1 Like
2

I think you can do it with logstash.
It has a plugin for mongodb as input, and elasticsearch can be configured as output.
The other way is what you want to avoid I guess - in this case you can create a small application which loads records from mongodb and inserts them into elasticsearch through the bulk API.

3

Thanks. These do seem to be the 2 best options.
We're not currently using logstash at all, but I have been considering introducing it to our infrastructure for other purposes, this would be "better" solution. However, developing a small application would probably be the quickest and easiest for now.
W.

4

What about mongo-connector?

It currently does not work with Elasticsearch 2.x though they are working on it.

The logstash input plugin is currently not listed in the logstash docs. Why is this?

5

I looked at the mongo-connector, just wasn't sure if this was going to be particularly future proof. It's definitely another option though, thanks!

It's doing doing what the river plugin does, and has GridFS support, which is good.

Regarding the logstash input plugin for MongoDB, there isn't an official one, only for output.

1 Like
6

Hi Warren

Thanks for asking this question. Can you let me know which solution looks better and have you started implementing this?

Thanks
Sri Harsha

7

Hi Sri,

I've been experimenting with the mongo-connector, seems to work quite well. I have it running and indexing our data in the simplest possible way.

We've built a basic sandbox on top of our elastic search server (because we're not running Kibanna) to validate what's being indexed.

The next step is finalise our API for access controlled searches, and potentially change slightly what we're indexing so that our API implementation is as clean as it can possibly be.

Opted for the mongo-connector in the end because I didn't want to re-invent any wheels. Hoping there will be an official MongoDB input plugin soon.

Kind Regards
Warren.

8

I wrote a simple tool. Maybe it will work.

https://github.com/indatawetrust/mongo-to-elastic