MongoDB River Plugin for ElasticSearch


(Richard Louapre) #1

Hi,

You can find [0] a first implementation of MongoDB plugin for ES.

This version has the following features:

  • Monitoring MongoDB using oplog.rs collection.
  • Support for GridFS.

Please check the wiki [1] for more details.

[0] - https://github.com/richardwilly98/elasticsearch-river-mongodb
[1] - https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki

Thanks,
Richard.


(Shay Banon) #2

Thanks!

On Monday, January 30, 2012 at 6:34 PM, Richard Louapre wrote:

Hi,

You can find [0] a first implementation of MongoDB plugin for ES.

This version has the following features:

  • Monitoring MongoDB using oplog.rs collection.
  • Support for GridFS.

Please check the wiki [1] for more details.

[0] - https://github.com/richardwilly98/elasticsearch-river-mongodb
[1] - https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki

Thanks,
Richard.


(Yammi) #3

WHOOT!

Thank god this exists.

On Jan 30, 11:34 am, Richard Louapre richard.loua...@gmail.com
wrote:

Hi,

You can find [0] a first implementation of MongoDB plugin for ES.

This version has the following features:

  • Monitoring MongoDB using oplog.rs collection.
  • Support for GridFS.

Please check the wiki [1] for more details.

[0] -https://github.com/richardwilly98/elasticsearch-river-mongodb
[1] -https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki

Thanks,
Richard.


(Arny) #4

Hi,

is the river tracking updates on documents inside mongodb as well?

On 30 Jan., 17:34, Richard Louapre richard.loua...@gmail.com wrote:

Hi,

You can find [0] a first implementation ofMongoDBplugin for ES.

This version has the following features:

  • MonitoringMongoDBusing oplog.rs collection.
  • Support for GridFS.

Please check the wiki [1] for more details.

[0] -https://github.com/richardwilly98/elasticsearch-river-mongodb
[1] -https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki

Thanks,
Richard.


(Arny) #5

I don't know if I'm missing something, but I can't get this to work.

  1. I've set up an replica with just 1 node for test purposes.
  2. Installed the river just like in docs stated.
  3. Add an document to mongodb

But, nothing happens?
I don't see any exceptions or anything that would show a failure.

I'm using Java API to set up the river:
this.esNode = NodeBuilder.nodeBuilder()
.settings(ImmutableSettings.settingsBuilder().put("gateway.type",
"local")).node();

	// mongodb river index
	XContentBuilder json;
	try {
		json = jsonBuilder()
				.startObject()
					.field("type", "mongodb")
					.field("mongodb").startObject()
						.field("db", "test")
						.field("collection", "test")
					.endObject()
					.field("index").startObject()
						.field("name", "testname")
						.field("type", "testtype")
					.endObject()
				.endObject();

		this.esNode.client().prepareIndex("_river", "mongodb",

"_meta").setSource(json).execute();

On Jan 30, 5:34 pm, Richard Louapre richard.loua...@gmail.com
wrote:

Hi,

You can find [0] a first implementation of MongoDB plugin for ES.

This version has the following features:

  • Monitoring MongoDB using oplog.rs collection.
  • Support for GridFS.

Please check the wiki [1] for more details.

[0] -https://github.com/richardwilly98/elasticsearch-river-mongodb
[1] -https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki

Thanks,
Richard.


(Arny) #6

Ok, my bad.
I've switched to ES RC1 while the plugin was still using 0.18.5.
So I've updated all the changes and now everything seems to work.

Thanks for the river :wink:

On Feb 8, 9:06 pm, Arny arny...@googlemail.com wrote:

I don't know if I'm missing something, but I can't get this to work.

  1. I've set up an replica with just 1 node for test purposes.
  2. Installed the river just like in docs stated.
  3. Add an document to mongodb

But, nothing happens?
I don't see any exceptions or anything that would show a failure.

I'm using Java API to set up the river:
this.esNode = NodeBuilder.nodeBuilder()
.settings(ImmutableSettings.settingsBuilder().put("gateway.type",
"local")).node();

            // mongodb river index
            XContentBuilder json;
            try {
                    json = jsonBuilder()
                                    .startObject()
                                            .field("type", "mongodb")
                                            .field("mongodb").startObject()
                                                    .field("db", "test")
                                                    .field("collection", "test")
                                            .endObject()
                                            .field("index").startObject()
                                                    .field("name", "testname")
                                                    .field("type", "testtype")
                                            .endObject()
                                    .endObject();

                    this.esNode.client().prepareIndex("_river", "mongodb",

"_meta").setSource(json).execute();

On Jan 30, 5:34 pm, Richard Louapre richard.loua...@gmail.com
wrote:

Hi,

You can find [0] a first implementation of MongoDB plugin for ES.

This version has the following features:

  • Monitoring MongoDB using oplog.rs collection.
  • Support for GridFS.

Please check the wiki [1] for more details.

[0] -https://github.com/richardwilly98/elasticsearch-river-mongodb
[1] -https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki

Thanks,
Richard.


(Arny) #7

@Richard

seems like partial updates, like increment are not supported now?
Logs on this

Cannot get object id. Skip the current item: [{$set={ "userLikesCount" : 1}, _id=null}]
Oplog do not contain _id inside "o", but there is "o2" which contains
the _id of the document.

Do you think you could fix this?
Thanks

On Jan 30, 5:34 pm, Richard Louapre richard.loua...@gmail.com
wrote:

Hi,

You can find [0] a first implementation of MongoDB plugin for ES.

This version has the following features:

  • Monitoring MongoDB using oplog.rs collection.
  • Support for GridFS.

Please check the wiki [1] for more details.

[0] -https://github.com/richardwilly98/elasticsearch-river-mongodb
[1] -https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki

Thanks,
Richard.


(Richard Louapre) #8

Hi,

I have created a new issue [0]. I will look at it.

[0] -

Ciao,
Richard.

On Friday, February 17, 2012 1:06:01 PM UTC-5, Arny wrote:

@Richard

seems like partial updates, like increment are not supported now?
Logs on this

Cannot get object id. Skip the current item: [{$set={ "userLikesCount" :
1}, _id=null}]
Oplog do not contain _id inside "o", but there is "o2" which contains
the _id of the document.

Do you think you could fix this?
Thanks

On Jan 30, 5:34 pm, Richard Louapre richard.loua...@gmail.com
wrote:

Hi,

You can find [0] a first implementation of MongoDB plugin for ES.

This version has the following features:

  • Monitoring MongoDB using oplog.rs collection.
  • Support for GridFS.

Please check the wiki [1] for more details.

[0] -https://github.com/richardwilly98/elasticsearch-river-mongodb
[1] -https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki

Thanks,
Richard.


(orenmazor) #9

awesome! I'm excited to stop using my homerolled oplog follower
(https://github.com/orenmazor/elastic-search-loves-mongo …slightly out of
date now).

can I use this to give weight to different ops? my current problem is that
my delete and index activity is pretty equally high, and I'd like to
prioritze indexing over delete (i.e. I'm currently modifying my oplog
follower to have an internal queue with qos)

On Monday, January 30, 2012 11:34:24 AM UTC-5, Richard Louapre wrote:

Hi,

You can find [0] a first implementation of MongoDB plugin for ES.

This version has the following features:

  • Monitoring MongoDB using oplog.rs collection.
  • Support for GridFS.

Please check the wiki [1] for more details.

[0] - https://github.com/richardwilly98/elasticsearch-river-mongodb
[1] - https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki

Thanks,
Richard.


(system) #10