Per MongoDB shard per ElasticSearch cluster?

Elastic Stack Elasticsearch
#1

Hello all, I'm trying to incorporate ElasticSearch with our existing
MongoDB database.
Currently we have a mongo collection separated to three mongo shards.
Does that mean I have to use three ElasticSearch clusters for each of them?
Is there a simpler practice to just use a single ElasticSearch cluster only?

--

#2

Hello Liang-Yu,

On Wed, Oct 24, 2012 at 6:38 AM, Liang-Yu Chou fishlet0528@gmail.com wrote:

Hello all, I'm trying to incorporate ElasticSearch with our existing MongoDB
database.
Currently we have a mongo collection separated to three mongo shards.
Does that mean I have to use three ElasticSearch clusters for each of them?

You shouldn't need 3 clusters. You might need more than one
Elasticsearch server to handle the load, but that depends on a lot of
factors, mainly about how your data, queries and hardware look like.

Is there a simpler practice to just use a single ElasticSearch cluster only?

The easiest way to index data from your MongoDB cluster to an
Elasticsearch cluster is using the MongoDB river:

There's a nice Wiki there as well, with instructions:

Best regards,
Radu

http://sematext.com/ -- ElasticSearch -- Solr -- Lucene

--

#3

Thank you Radu. So you mean I can setup an ElasticSearch cluster and let it
pull data
from, in my case, three mongo shards, and write into one single index?
If yes, that explains a lot. I previously thought I can only specify one
data source to pull from
for each ElasticSearch index...

Radu Gheorghe於 2012年10月24日星期三UTC+8下午8時25分03秒寫道：

Hello Liang-Yu,

On Wed, Oct 24, 2012 at 6:38 AM, Liang-Yu Chou <fishl...@gmail.com<javascript:>>
wrote:

Hello all, I'm trying to incorporate ElasticSearch with our existing
MongoDB
database.
Currently we have a mongo collection separated to three mongo shards.
Does that mean I have to use three ElasticSearch clusters for each of
them?

You shouldn't need 3 clusters. You might need more than one
Elasticsearch server to handle the load, but that depends on a lot of
factors, mainly about how your data, queries and hardware look like.

Is there a simpler practice to just use a single ElasticSearch cluster
only?

The easiest way to index data from your MongoDB cluster to an
Elasticsearch cluster is using the MongoDB river:
https://github.com/richardwilly98/elasticsearch-river-mongodb

There's a nice Wiki there as well, with instructions:
https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki

Best regards,
Radu

http://sematext.com/ -- ElasticSearch -- Solr -- Lucene

--

#4

Hello,

I haven't tested this specific setup, but I don't see why you
couldn't. You can define multiple rivers, and each river would have
its own configuration. As for indices, they don't have limitations in
terms of data source.

Best regards,
Radu

http://sematext.com/ -- ElasticSearch -- Solr -- Lucene

On Thu, Oct 25, 2012 at 9:50 AM, Jerry Chou fishlet0528@gmail.com wrote:

Thank you Radu. So you mean I can setup an ElasticSearch cluster and let it
pull data
from, in my case, three mongo shards, and write into one single index?
If yes, that explains a lot. I previously thought I can only specify one
data source to pull from
for each ElasticSearch index...

Radu Gheorghe於 2012年10月24日星期三UTC+8下午8時25分03秒寫道：

Hello Liang-Yu,

On Wed, Oct 24, 2012 at 6:38 AM, Liang-Yu Chou fishl...@gmail.com wrote:

Hello all, I'm trying to incorporate ElasticSearch with our existing
MongoDB
database.
Currently we have a mongo collection separated to three mongo shards.
Does that mean I have to use three ElasticSearch clusters for each of
them?

You shouldn't need 3 clusters. You might need more than one
Elasticsearch server to handle the load, but that depends on a lot of
factors, mainly about how your data, queries and hardware look like.

Is there a simpler practice to just use a single ElasticSearch cluster
only?

The easiest way to index data from your MongoDB cluster to an
Elasticsearch cluster is using the MongoDB river:
https://github.com/richardwilly98/elasticsearch-river-mongodb

There's a nice Wiki there as well, with instructions:
https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki

Best regards,
Radu

http://sematext.com/ -- ElasticSearch -- Solr -- Lucene

--

--

#5

Hi All,

I have configured elasticsearch mongodb river on mongodb shard cluster to index data. Have 3 shareds and configured river on each shard(mean have 3 indexes in single ES cluster). Here i facing issue, when data migration happening among mongodb shards those changes not reflecting in elasticsearch.

Can anybody please help me.

Or one river configuration would on complete mongo shard cluster(on mongose)?

Thanks,
Yesu

#6