ES with Mongodb

Elastic Stack Elasticsearch
1

Hi,

Has anyone integrated ES with mongodb. I would really appreciate if
you share your experience and how you have done it.

I am exploring options to use ES for full text search for some of my
data stored in Mongo.
Is there any easy way to update ES indexes as and when mongodb gets
updated? Just so that I can avoid duplicating the data.

Thanks
Gautam

2

Hi,

There were some questions about it some time ago, maybe people who have
done it can jump in. The way that I see it, you will need to integrate the
sync between the two on the client side. For example, everytime you change
something on mongo, apply the same change to ES (either directly or through
a message queue).

I don't see a way that ES can do it automatically by querying mongo (as
it does with couchdb and the _changes stream). But, if someone has an idea,
I would be happy to hear it and maybe we can implement it.

-shay.banon

On Sun, Nov 28, 2010 at 1:02 PM, Gautam mrgautamsam@gmail.com wrote:

Hi,

Has anyone integrated ES with mongodb. I would really appreciate if
you share your experience and how you have done it.

I am exploring options to use ES for full text search for some of my
data stored in Mongo.
Is there any easy way to update ES indexes as and when mongodb gets
updated? Just so that I can avoid duplicating the data.

Thanks
Gautam

3

Thanks Shay for your response. Much appreciated.
Yup, that is a way to do it.
But was looking for a much more seamless integration.

It would have been great if I could have specified something like (using a
push mechanism):

  1. Watch Collection A and B
  2. Index a selected set of attributes and subdocuments of a particular
    Collection.

And that's it.
Then onwards any change in these collections ES gets notified and the job is
done.

All these can be done in a wrapper layer as well, but would have ideal
through a mongo push.

Thanks
Gautam

On Mon, Nov 29, 2010 at 4:36 AM, Shay Banon shay.banon@elasticsearch.comwrote:

Hi,

There were some questions about it some time ago, maybe people who have
done it can jump in. The way that I see it, you will need to integrate the
sync between the two on the client side. For example, everytime you change
something on mongo, apply the same change to ES (either directly or through
a message queue).

I don't see a way that ES can do it automatically by querying mongo (as
it does with couchdb and the _changes stream). But, if someone has an idea,
I would be happy to hear it and maybe we can implement it.

-shay.banon

On Sun, Nov 28, 2010 at 1:02 PM, Gautam mrgautamsam@gmail.com wrote:

Hi,

Has anyone integrated ES with mongodb. I would really appreciate if
you share your experience and how you have done it.

I am exploring options to use ES for full text search for some of my
data stored in Mongo.
Is there any easy way to update ES indexes as and when mongodb gets
updated? Just so that I can avoid duplicating the data.

Thanks
Gautam

4

Agreed, though I am not familiar with a mongo push feature, do they have it?

On Mon, Nov 29, 2010 at 10:02 AM, Gautam Mr mrgautamsam@gmail.com wrote:

Thanks Shay for your response. Much appreciated.
Yup, that is a way to do it.
But was looking for a much more seamless integration.

It would have been great if I could have specified something like (using a
push mechanism):

  1. Watch Collection A and B
  2. Index a selected set of attributes and subdocuments of a particular
    Collection.

And that's it.
Then onwards any change in these collections ES gets notified and the job
is done.

All these can be done in a wrapper layer as well, but would have ideal
through a mongo push.

Thanks
Gautam

On Mon, Nov 29, 2010 at 4:36 AM, Shay Banon shay.banon@elasticsearch.comwrote:

Hi,

There were some questions about it some time ago, maybe people who have
done it can jump in. The way that I see it, you will need to integrate the
sync between the two on the client side. For example, everytime you change
something on mongo, apply the same change to ES (either directly or through
a message queue).

I don't see a way that ES can do it automatically by querying mongo (as
it does with couchdb and the _changes stream). But, if someone has an idea,
I would be happy to hear it and maybe we can implement it.

-shay.banon

On Sun, Nov 28, 2010 at 1:02 PM, Gautam mrgautamsam@gmail.com wrote:

Hi,

Has anyone integrated ES with mongodb. I would really appreciate if
you share your experience and how you have done it.

I am exploring options to use ES for full text search for some of my
data stored in Mongo.
Is there any easy way to update ES indexes as and when mongodb gets
updated? Just so that I can avoid duplicating the data.

Thanks
Gautam

5

Have sent a query to Mongo team, will let you know when I hear back.

On Mon, Nov 29, 2010 at 3:36 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Agreed, though I am not familiar with a mongo push feature, do they have
it?

On Mon, Nov 29, 2010 at 10:02 AM, Gautam Mr mrgautamsam@gmail.com wrote:

Thanks Shay for your response. Much appreciated.
Yup, that is a way to do it.
But was looking for a much more seamless integration.

It would have been great if I could have specified something like (using a
push mechanism):

  1. Watch Collection A and B
  2. Index a selected set of attributes and subdocuments of a particular
    Collection.

And that's it.
Then onwards any change in these collections ES gets notified and the job
is done.

All these can be done in a wrapper layer as well, but would have ideal
through a mongo push.

Thanks
Gautam

On Mon, Nov 29, 2010 at 4:36 AM, Shay Banon <shay.banon@elasticsearch.com

wrote:

Hi,

There were some questions about it some time ago, maybe people who have
done it can jump in. The way that I see it, you will need to integrate the
sync between the two on the client side. For example, everytime you change
something on mongo, apply the same change to ES (either directly or through
a message queue).

I don't see a way that ES can do it automatically by querying mongo
(as it does with couchdb and the _changes stream). But, if someone has an
idea, I would be happy to hear it and maybe we can implement it.

-shay.banon

On Sun, Nov 28, 2010 at 1:02 PM, Gautam mrgautamsam@gmail.com wrote:

Hi,

Has anyone integrated ES with mongodb. I would really appreciate if
you share your experience and how you have done it.

I am exploring options to use ES for full text search for some of my
data stored in Mongo.
Is there any easy way to update ES indexes as and when mongodb gets
updated? Just so that I can avoid duplicating the data.

Thanks
Gautam