Newbie trying to setup elastic search with mongodb and nodejs

nik_jain · May 16, 2013, 6:15pm

I have followed the steps in this guide here: https://coderwall.com/p/sy1qcw

This is what I am trying to update with.

#!/bin/sh
curl -XPUT "localhost:9200/_river/some_db/_meta" -d '
{
"type": "mongodb",
"mongodb": {
"servers": [
{ "host": "127.0.0.1", "port": 27017 }
],
"options": { "secondary_read_preference": true },
"db": "some_db",
"collection": "twitter"
},
"index": {
"name": "text",
"type": "string"
}

}'.

some_db - is the database name
text - is the key of the property that I want to index.

But this throws this error:
{"error":"RemoteTransportException[[Goblyn][inet[/192.168.0.101:9302]][index]];
nested: IndexFailedEngineException[[_river][0] Index failed for
[some_db#_meta]]; nested: AlreadyClosedException[this IndexWriter is
closed]; ","status":500}

I have no idea how to proceed further.
I am using
ubuntu 12.10
elasticsarch .90
elasticsearch-mapper-attachments-1.6.0
elasticsearch-river-mongodb-master
node-js -
mongodb-10gen package.

HELP!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

nik_jain · May 16, 2013, 6:35pm

curl -XPUT 'localhost:9200/_river/my_river/_meta' -d '{
"type" : "dummy"
}'

this also throws the same error.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

nik_jain · May 16, 2013, 6:47pm

OK I executed :
curl -XPOST 'localhost:9200/_river/_open'

and then reran the command which gave the following error:
{"error":"UnavailableShardsException[[_river][0] [2] shardIt, [0] active :
Timeout waiting for [1m], request: index {[_river][some_db][_meta],
source[\n{\n "type": "mongodb",\n "mongodb": {

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Clinton_Gormley · May 16, 2013, 7:43pm

Hi Nik

I don't recommend using the mongodb river. It is buggy and incomplete and
not supported by Elasticsearch.

Instead, I'd just push your documents into Elasticsearch using the bulk
API: Elasticsearch Platform — Find real-time answers at scale | Elastic

You'll save yourself a lot of time

clint

On 16 May 2013 20:47, nik.jain@gmail.com wrote:

OK I executed :
curl -XPOST 'localhost:9200/_river/_open'

and then reran the command which gave the following error:
{"error":"UnavailableShardsException[[_river][0] [2] shardIt, [0] active
: Timeout waiting for [1m], request: index {[_river][some_db][_meta],
source[\n{\n "type": "mongodb",\n "mongodb": {

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

nik_jain · May 16, 2013, 9:09pm

Hey
Thanks much for the reply. Could you please give me a quick opinion on the
jdbc river plugin? is it stable ?

Thanks

On Friday, May 17, 2013 1:13:59 AM UTC+5:30, Clinton Gormley wrote:

Hi Nik

I don't recommend using the mongodb river. It is buggy and incomplete and
not supported by Elasticsearch.

Instead, I'd just push your documents into Elasticsearch using the bulk
API: Elasticsearch Platform — Find real-time answers at scale | Elastic

You'll save yourself a lot of time

clint

On 16 May 2013 20:47, <nik....@gmail.com <javascript:>> wrote:

OK I executed :
curl -XPOST 'localhost:9200/_river/_open'

and then reran the command which gave the following error:
{"error":"UnavailableShardsException[[_river][0] [2] shardIt, [0] active
: Timeout waiting for [1m], request: index {[_river][some_db][_meta],
source[\n{\n "type": "mongodb",\n "mongodb": {

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · May 17, 2013, 12:00am

Where are your source docs?
In a SQL DB or in MongoDB or both?

IMHO, JDBC river works fine but I prefer that Jörg answers directly to this question as he created the plugin.

That said, you should be aware of that JDBC river is pulling docs every x minute. It's not a near real time indexing job.

Let me say that in an other way. If you have a service layer in your application and from the service layer you persist your data in any SQL database, just send it at the same time to Elasticsearch for indexing. That way, you will have a finer control of what you are indexing and it will be done near real time…

My 2 cents.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 16 mai 2013 à 23:09, nik.jain@gmail.com a écrit :

Hey
Thanks much for the reply. Could you please give me a quick opinion on the jdbc river plugin? is it stable ?

Thanks

On Friday, May 17, 2013 1:13:59 AM UTC+5:30, Clinton Gormley wrote:
Hi Nik

I don't recommend using the mongodb river. It is buggy and incomplete and not supported by Elasticsearch.

Instead, I'd just push your documents into Elasticsearch using the bulk API: Elasticsearch Platform — Find real-time answers at scale | Elastic

You'll save yourself a lot of time

clint

On 16 May 2013 20:47, nik....@gmail.com wrote:
OK I executed :
curl -XPOST 'localhost:9200/_river/_open'

and then reran the command which gave the following error:
{"error":"UnavailableShardsException[[_river][0] [2] shardIt, [0] active : Timeout waiting for [1m], request: index {[_river][some_db][_meta], source[\n{\n "type": "mongodb",\n "mongodb": {

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

nik_jain · May 17, 2013, 8:39am

Thanks david,
Is there any solid working full example of using the bulk api ? THe
docs are just a reference and difficult for me to comprehend. I
understand of course that a simple curl request would deal with
insertion of data into the index.

On May 17, 5:00 am, David Pilato da...@pilato.fr wrote:

Where are your source docs?
In a SQL DB or in MongoDB or both?

IMHO, JDBC river works fine but I prefer that Jörg answers directly to this question as he created the plugin.

That said, you should be aware of that JDBC river is pulling docs every x minute. It's not a near real time indexing job.

Let me say that in an other way. If you have a service layer in your application and from the service layer you persist your data in any SQL database, just send it at the same time to Elasticsearch for indexing. That way, you will have a finer control of what you are indexing and it will be done near real time…

My 2 cents.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 16 mai 2013 à 23:09, nik.j...@gmail.com a écrit :

Hey
Thanks much for the reply. Could you please give me a quick opinion on the jdbc river plugin? is it stable ?

Thanks

On Friday, May 17, 2013 1:13:59 AM UTC+5:30, Clinton Gormley wrote:
Hi Nik

I don't recommend using the mongodb river. It is buggy and incomplete and not supported by Elasticsearch.

Instead, I'd just push your documents into Elasticsearch using the bulk API:Elasticsearch Platform — Find real-time answers at scale | Elastic

You'll save yourself a lot of time

clint

On 16 May 2013 20:47, nik....@gmail.com wrote:
OK I executed :
curl -XPOST 'localhost:9200/_river/_open'

and then reran the command which gave the following error:
{"error":"UnavailableShardsException[[_river][0] [2] shardIt, [0] active : Timeout waiting for [1m], request: index {[_river][some_db][_meta], source[\n{\n "type": "mongodb",\n "mongodb": {

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visithttps://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visithttps://groups.google.com/groups/opt_out.

smime.p7s
1KViewDownload

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · May 17, 2013, 8:52am

Using NodeJS? I don't know. Are you using any specific Client?

In Java: Elasticsearch Platform — Find real-time answers at scale | Elastic

Using CURL, the example here works: Elasticsearch Platform — Find real-time answers at scale | Elastic

If you don't succeed in using it, please provide a full curl recreation of what you tried so far.

HTH

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 17 mai 2013 à 10:39, Goddamnit nik.jain@gmail.com a écrit :

Thanks david,
Is there any solid working full example of using the bulk api ? THe
docs are just a reference and difficult for me to comprehend. I
understand of course that a simple curl request would deal with
insertion of data into the index.

On May 17, 5:00 am, David Pilato da...@pilato.fr wrote:

Where are your source docs?
In a SQL DB or in MongoDB or both?

IMHO, JDBC river works fine but I prefer that Jörg answers directly to this question as he created the plugin.

That said, you should be aware of that JDBC river is pulling docs every x minute. It's not a near real time indexing job.

Let me say that in an other way. If you have a service layer in your application and from the service layer you persist your data in any SQL database, just send it at the same time to Elasticsearch for indexing. That way, you will have a finer control of what you are indexing and it will be done near real time…

My 2 cents.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 16 mai 2013 à 23:09, nik.j...@gmail.com a écrit :

Hey
Thanks much for the reply. Could you please give me a quick opinion on the jdbc river plugin? is it stable ?

Thanks

On Friday, May 17, 2013 1:13:59 AM UTC+5:30, Clinton Gormley wrote:
Hi Nik

I don't recommend using the mongodb river. It is buggy and incomplete and not supported by Elasticsearch.

Instead, I'd just push your documents into Elasticsearch using the bulk API:Elasticsearch Platform — Find real-time answers at scale | Elastic

You'll save yourself a lot of time

clint

On 16 May 2013 20:47, nik....@gmail.com wrote:
OK I executed :
curl -XPOST 'localhost:9200/_river/_open'

and then reran the command which gave the following error:
{"error":"UnavailableShardsException[[_river][0] [2] shardIt, [0] active : Timeout waiting for [1m], request: index {[_river][some_db][_meta], source[\n{\n "type": "mongodb",\n "mongodb": {

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visithttps://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visithttps://groups.google.com/groups/opt_out.

smime.p7s
1KViewDownload

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Ivan · May 17, 2013, 4:01pm

If you are using the Java client, then looking at the source of any of the
existing indexing rivers will provide some insight on how bulk indexing
works. The rivers are singleton applications that use the same APIs as any
other client. The official Elasticsearch rivers should all be using the
BulkProcessor class, which is probably easier to understand and more
efficient.

--
Ivan

On Fri, May 17, 2013 at 1:39 AM, Goddamnit nik.jain@gmail.com wrote:

Thanks david,
Is there any solid working full example of using the bulk api ? THe
docs are just a reference and difficult for me to comprehend. I
understand of course that a simple curl request would deal with
insertion of data into the index.

On May 17, 5:00 am, David Pilato da...@pilato.fr wrote:

Where are your source docs?
In a SQL DB or in MongoDB or both?

IMHO, JDBC river works fine but I prefer that Jörg answers directly to
this question as he created the plugin.

That said, you should be aware of that JDBC river is pulling docs every
x minute. It's not a near real time indexing job.

Let me say that in an other way. If you have a service layer in your
application and from the service layer you persist your data in any SQL
database, just send it at the same time to Elasticsearch for indexing. That
way, you will have a finer control of what you are indexing and it will be
done near real time…

My 2 cents.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 16 mai 2013 à 23:09, nik.j...@gmail.com a écrit :

Hey
Thanks much for the reply. Could you please give me a quick opinion on
the jdbc river plugin? is it stable ?

Thanks

On Friday, May 17, 2013 1:13:59 AM UTC+5:30, Clinton Gormley wrote:
Hi Nik

I don't recommend using the mongodb river. It is buggy and incomplete
and not supported by Elasticsearch.

Instead, I'd just push your documents into Elasticsearch using the
bulk API:Elasticsearch Platform — Find real-time answers at scale | Elastic

You'll save yourself a lot of time

clint

On 16 May 2013 20:47, nik....@gmail.com wrote:
OK I executed :
curl -XPOST 'localhost:9200/_river/_open'

and then reran the command which gave the following error:
{"error":"UnavailableShardsException[[_river][0] [2] shardIt, [0]
active : Timeout waiting for [1m], request: index
{[_river][some_db][_meta], source[\n{\n "type": "mongodb",\n
"mongodb": {

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visithttps://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visithttps://groups.google.com/groups/opt_out.

smime.p7s
1KViewDownload

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.