MongoDB River Plugin for ElasticSearch

Richard_Louapre · January 30, 2012, 4:34pm

Hi,

You can find [0] a first implementation of MongoDB plugin for ES.

This version has the following features:

Monitoring MongoDB using oplog.rs collection.
Support for GridFS.

Please check the wiki [1] for more details.

[0] - https://github.com/richardwilly98/elasticsearch-river-mongodb
[1] - https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki

Thanks,
Richard.

kimchy · January 31, 2012, 3:50pm

Thanks!

On Monday, January 30, 2012 at 6:34 PM, Richard Louapre wrote:

Hi,

You can find [0] a first implementation of MongoDB plugin for ES.

This version has the following features:

Monitoring MongoDB using oplog.rs collection.

Support for GridFS.

Please check the wiki [1] for more details.

[0] - GitHub - richardwilly98/elasticsearch-river-mongodb: MongoDB River Plugin for ElasticSearch
[1] - Home · richardwilly98/elasticsearch-river-mongodb Wiki · GitHub

Thanks,
Richard.

Yammi · February 1, 2012, 11:42am

WHOOT!

Thank god this exists.

On Jan 30, 11:34 am, Richard Louapre richard.loua...@gmail.com
wrote:

Hi,

You can find [0] a first implementation of MongoDB plugin for ES.

This version has the following features:

Monitoring MongoDB using oplog.rs collection.

Support for GridFS.

Please check the wiki [1] for more details.

[0] -GitHub - richardwilly98/elasticsearch-river-mongodb: MongoDB River Plugin for ElasticSearch
[1] -Home · richardwilly98/elasticsearch-river-mongodb Wiki · GitHub

Thanks,
Richard.

Arny · February 8, 2012, 4:21am

Hi,

is the river tracking updates on documents inside mongodb as well?

On 30 Jan., 17:34, Richard Louapre richard.loua...@gmail.com wrote:

Hi,

You can find [0] a first implementation ofMongoDBplugin for ES.

This version has the following features:

MonitoringMongoDBusing oplog.rs collection.

Support for GridFS.

Please check the wiki [1] for more details.

[0] -GitHub - richardwilly98/elasticsearch-river-mongodb: MongoDB River Plugin for ElasticSearch
[1] -Home · richardwilly98/elasticsearch-river-mongodb Wiki · GitHub

Thanks,
Richard.

Arny · February 8, 2012, 8:06pm

I don't know if I'm missing something, but I can't get this to work.

I've set up an replica with just 1 node for test purposes.
Installed the river just like in docs stated.
Add an document to mongodb

But, nothing happens?
I don't see any exceptions or anything that would show a failure.

I'm using Java API to set up the river:
this.esNode = NodeBuilder.nodeBuilder()
.settings(ImmutableSettings.settingsBuilder().put("gateway.type",
"local")).node();

	// mongodb river index
	XContentBuilder json;
	try {
		json = jsonBuilder()
				.startObject()
					.field("type", "mongodb")
					.field("mongodb").startObject()
						.field("db", "test")
						.field("collection", "test")
					.endObject()
					.field("index").startObject()
						.field("name", "testname")
						.field("type", "testtype")
					.endObject()
				.endObject();

		this.esNode.client().prepareIndex("_river", "mongodb",

"_meta").setSource(json).execute();

On Jan 30, 5:34 pm, Richard Louapre richard.loua...@gmail.com
wrote:

Hi,

You can find [0] a first implementation of MongoDB plugin for ES.

This version has the following features:

Monitoring MongoDB using oplog.rs collection.

Support for GridFS.

Please check the wiki [1] for more details.

[0] -GitHub - richardwilly98/elasticsearch-river-mongodb: MongoDB River Plugin for ElasticSearch
[1] -Home · richardwilly98/elasticsearch-river-mongodb Wiki · GitHub

Thanks,
Richard.

Arny · February 8, 2012, 8:23pm

Ok, my bad.
I've switched to ES RC1 while the plugin was still using 0.18.5.
So I've updated all the changes and now everything seems to work.

Thanks for the river

On Feb 8, 9:06 pm, Arny arny...@googlemail.com wrote:

I don't know if I'm missing something, but I can't get this to work.

I've set up an replica with just 1 node for test purposes.

Installed the river just like in docs stated.

Add an document to mongodb

But, nothing happens?
I don't see any exceptions or anything that would show a failure.

I'm using Java API to set up the river:
this.esNode = NodeBuilder.nodeBuilder()
.settings(ImmutableSettings.settingsBuilder().put("gateway.type",
"local")).node();
            // mongodb river index
            XContentBuilder json;
            try {
                    json = jsonBuilder()
                                    .startObject()
                                            .field("type", "mongodb")
                                            .field("mongodb").startObject()
                                                    .field("db", "test")
                                                    .field("collection", "test")
                                            .endObject()
                                            .field("index").startObject()
                                                    .field("name", "testname")
                                                    .field("type", "testtype")
                                            .endObject()
                                    .endObject();

                    this.esNode.client().prepareIndex("_river", "mongodb",
"_meta").setSource(json).execute();

On Jan 30, 5:34 pm, Richard Louapre richard.loua...@gmail.com
wrote:

Hi,

You can find [0] a first implementation of MongoDB plugin for ES.

This version has the following features:

Monitoring MongoDB using oplog.rs collection.

Support for GridFS.

Please check the wiki [1] for more details.

[0] -GitHub - richardwilly98/elasticsearch-river-mongodb: MongoDB River Plugin for ElasticSearch
[1] -Home · richardwilly98/elasticsearch-river-mongodb Wiki · GitHub

Thanks,
Richard.

Arny · February 17, 2012, 6:06pm

@Richard

seems like partial updates, like increment are not supported now?
Logs on this

Cannot get object id. Skip the current item: [{$set={ "userLikesCount" : 1}, _id=null}]
Oplog do not contain _id inside "o", but there is "o2" which contains
the _id of the document.

Do you think you could fix this?
Thanks

On Jan 30, 5:34 pm, Richard Louapre richard.loua...@gmail.com
wrote:

Hi,

You can find [0] a first implementation of MongoDB plugin for ES.

This version has the following features:

Monitoring MongoDB using oplog.rs collection.

Support for GridFS.

Please check the wiki [1] for more details.

[0] -GitHub - richardwilly98/elasticsearch-river-mongodb: MongoDB River Plugin for ElasticSearch
[1] -Home · richardwilly98/elasticsearch-river-mongodb Wiki · GitHub

Thanks,
Richard.

Richard_Louapre · March 4, 2012, 11:47am

Hi,

I have created a new issue [0]. I will look at it.

[0] -

Ciao,
Richard.

On Friday, February 17, 2012 1:06:01 PM UTC-5, Arny wrote:

@Richard

seems like partial updates, like increment are not supported now?
Logs on this

Cannot get object id. Skip the current item: [{$set={ "userLikesCount" :
1}, _id=null}]
Oplog do not contain _id inside "o", but there is "o2" which contains
the _id of the document.

Do you think you could fix this?
Thanks

On Jan 30, 5:34 pm, Richard Louapre richard.loua...@gmail.com
wrote:

Hi,

You can find [0] a first implementation of MongoDB plugin for ES.

This version has the following features:

Monitoring MongoDB using oplog.rs collection.

Support for GridFS.

Please check the wiki [1] for more details.

[0] -GitHub - richardwilly98/elasticsearch-river-mongodb: MongoDB River Plugin for ElasticSearch
[1] -Home · richardwilly98/elasticsearch-river-mongodb Wiki · GitHub

Thanks,
Richard.

orenmazor · March 7, 2012, 9:26pm

awesome! I'm excited to stop using my homerolled oplog follower
(GitHub - orenmazor/elastic-search-loves-mongo …slightly out of
date now).

can I use this to give weight to different ops? my current problem is that
my delete and index activity is pretty equally high, and I'd like to
prioritze indexing over delete (i.e. I'm currently modifying my oplog
follower to have an internal queue with qos)

On Monday, January 30, 2012 11:34:24 AM UTC-5, Richard Louapre wrote:

Hi,

You can find [0] a first implementation of MongoDB plugin for ES.

This version has the following features:

Monitoring MongoDB using oplog.rs collection.

Support for GridFS.

Please check the wiki [1] for more details.

[0] - GitHub - richardwilly98/elasticsearch-river-mongodb: MongoDB River Plugin for ElasticSearch
[1] - Home · richardwilly98/elasticsearch-river-mongodb Wiki · GitHub

Thanks,
Richard.