# Newbie trying to setup elastic search with mongodb and nodejs **URL:** \ **Category:** Elasticsearch\ **Created:** [May 16, 2013, 6:15pm UTC](https://discuss.elastic.co/t/newbie-trying-to-setup-elastic-search-with-mongodb-and-nodejs/11996 "2013-05-16T18:15:54Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![nik\_jain](https://avatars.discourse-cdn.com/v4/letter/n/f05b48/32.png) [@nik\_jain](https://discuss.elastic.co/u/nik_jain)\ **Post date:** [May 16, 2013, 6:15pm UTC](https://discuss.elastic.co/t/newbie-trying-to-setup-elastic-search-with-mongodb-and-nodejs/11996/1 "2013-05-16T18:15:54Z")
I have followed the steps in this guide here: [https://coderwall.com/p/sy1qcw](https://coderwall.com/p/sy1qcw) This is what I am trying to update with. #!/bin/sh curl -XPUT "localhost:9200/\_river/some\_db/\_meta" -d ' { "type": "mongodb", "mongodb": { "servers": [ { "host": "127.0.0.1", "port": 27017 } ], "options": { "secondary\_read\_preference": true }, "db": "some\_db", "collection": "twitter" }, "index": { "name": "text", "type": "string" } }'. some\_db - is the database name text - is the key of the property that I want to index. But this throws this error: {"error":"RemoteTransportException[[Goblyn][inet[/192.168.0.101:9302]][index]]; nested: IndexFailedEngineException[[\_river][0] Index failed for [some\_db#\_meta]]; nested: AlreadyClosedException[this IndexWriter is closed]; ","status":500} I have no idea how to proceed further. I am using ubuntu 12.10 elasticsarch .90 elasticsearch-mapper-attachments-1.6.0 elasticsearch-river-mongodb-master node-js - mongodb-10gen package. HELP! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![nik\_jain](https://avatars.discourse-cdn.com/v4/letter/n/f05b48/32.png) [@nik\_jain](https://discuss.elastic.co/u/nik_jain)\ **Post date:** [May 16, 2013, 6:35pm UTC](https://discuss.elastic.co/t/newbie-trying-to-setup-elastic-search-with-mongodb-and-nodejs/11996/2 "2013-05-16T18:35:14Z")
curl -XPUT 'localhost:9200/\_river/my\_river/\_meta' -d '{ "type" : "dummy" }' - this also throws the same error. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![nik\_jain](https://avatars.discourse-cdn.com/v4/letter/n/f05b48/32.png) [@nik\_jain](https://discuss.elastic.co/u/nik_jain)\ **Post date:** [May 16, 2013, 6:47pm UTC](https://discuss.elastic.co/t/newbie-trying-to-setup-elastic-search-with-mongodb-and-nodejs/11996/3 "2013-05-16T18:47:46Z")
OK I executed : curl -XPOST 'localhost:9200/\_river/\_open' and then reran the command which gave the following error: {"error":"UnavailableShardsException[[\_river][0] [2] shardIt, [0] active : Timeout waiting for [1m], request: index {[\_river][some\_db][\_meta], source[\n{\n "type": "mongodb",\n "mongodb": { -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [May 16, 2013, 7:43pm UTC](https://discuss.elastic.co/t/newbie-trying-to-setup-elastic-search-with-mongodb-and-nodejs/11996/4 "2013-05-16T19:43:59Z")
Hi Nik I don't recommend using the mongodb river. It is buggy and incomplete and not supported by Elasticsearch. Instead, I'd just push your documents into Elasticsearch using the bulk API: [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/bulk/) You'll save yourself a lot of time 🙂 clint On 16 May 2013 20:47, [nik.jain@gmail.com](mailto:nik.jain@gmail.com) wrote: > OK I executed : > curl -XPOST 'localhost:9200/\_river/\_open' > > and then reran the command which gave the following error: > {"error":"UnavailableShardsException[[\_river][0] [2] shardIt, [0] active > : Timeout waiting for [1m], request: index {[\_river][some\_db][\_meta], > source[\n{\n "type": "mongodb",\n "mongodb": { > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![nik\_jain](https://avatars.discourse-cdn.com/v4/letter/n/f05b48/32.png) [@nik\_jain](https://discuss.elastic.co/u/nik_jain)\ **Post date:** [May 16, 2013, 9:09pm UTC](https://discuss.elastic.co/t/newbie-trying-to-setup-elastic-search-with-mongodb-and-nodejs/11996/5 "2013-05-16T21:09:41Z")
Hey Thanks much for the reply. Could you please give me a quick opinion on the jdbc river plugin? is it stable ? Thanks On Friday, May 17, 2013 1:13:59 AM UTC+5:30, Clinton Gormley wrote: > Hi Nik > > I don't recommend using the mongodb river. It is buggy and incomplete and > not supported by Elasticsearch. > > Instead, I'd just push your documents into Elasticsearch using the bulk > API: [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/bulk/) > > You'll save yourself a lot of time 🙂 > > clint > > On 16 May 2013 20:47, \<[nik....@gmail.com](mailto:nik....@gmail.com) \\> wrote: > > > OK I executed : > > curl -XPOST 'localhost:9200/\_river/\_open' > > > > and then reran the command which gave the following error: > > {"error":"UnavailableShardsException[[\_river][0] [2] shardIt, [0] active > > : Timeout waiting for [1m], request: index {[\_river][some\_db][\_meta], > > source[\n{\n "type": "mongodb",\n "mongodb": { > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 17, 2013, 12:00am UTC](https://discuss.elastic.co/t/newbie-trying-to-setup-elastic-search-with-mongodb-and-nodejs/11996/6 "2013-05-17T00:00:08Z")
Where are your source docs? In a SQL DB or in MongoDB or both? IMHO, JDBC river works fine but I prefer that Jörg answers directly to this question as he created the plugin. That said, you should be aware of that JDBC river is pulling docs every x minute. It's not a near real time indexing job. Let me say that in an other way. If you have a service layer in your application and from the service layer you persist your data in any SQL database, just send it at the same time to Elasticsearch for indexing. That way, you will have a finer control of what you are indexing and it will be done near real time… My 2 cents. -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr | @scrutmydocs Le 16 mai 2013 à 23:09, [nik.jain@gmail.com](mailto:nik.jain@gmail.com) a écrit : > Hey > Thanks much for the reply. Could you please give me a quick opinion on the jdbc river plugin? is it stable ? > > Thanks > > On Friday, May 17, 2013 1:13:59 AM UTC+5:30, Clinton Gormley wrote: > Hi Nik > > I don't recommend using the mongodb river. It is buggy and incomplete and not supported by Elasticsearch. > > Instead, I'd just push your documents into Elasticsearch using the bulk API: [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/bulk/) > > You'll save yourself a lot of time 🙂 > > clint > > On 16 May 2013 20:47, [nik....@gmail.com](mailto:nik....@gmail.com) wrote: > OK I executed : > curl -XPOST 'localhost:9200/\_river/\_open' > > and then reran the command which gave the following error: > {"error":"UnavailableShardsException[[\_river][0] [2] shardIt, [0] active : Timeout waiting for [1m], request: index {[\_river][some\_db][\_meta], source[\n{\n "type": "mongodb",\n "mongodb": { > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![nik\_jain](https://avatars.discourse-cdn.com/v4/letter/n/f05b48/32.png) [@nik\_jain](https://discuss.elastic.co/u/nik_jain)\ **Post date:** [May 17, 2013, 8:39am UTC](https://discuss.elastic.co/t/newbie-trying-to-setup-elastic-search-with-mongodb-and-nodejs/11996/7 "2013-05-17T08:39:24Z")
Thanks david, Is there any solid working full example of using the bulk api ? THe docs are just a reference and difficult for me to comprehend. I understand of course that a simple curl request would deal with insertion of data into the index. On May 17, 5:00 am, David Pilato [da...@pilato.fr](mailto:da...@pilato.fr) wrote: > Where are your source docs? > In a SQL DB or in MongoDB or both? > > IMHO, JDBC river works fine but I prefer that Jörg answers directly to this question as he created the plugin. > > That said, you should be aware of that JDBC river is pulling docs every x minute. It's not a near real time indexing job. > > Let me say that in an other way. If you have a service layer in your application and from the service layer you persist your data in any SQL database, just send it at the same time to Elasticsearch for indexing. That way, you will have a finer control of what you are indexing and it will be done near real time… > > My 2 cents. > > -- > David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) > @dadoonet | @elasticsearchfr | @scrutmydocs > > Le 16 mai 2013 à 23:09, [nik.j...@gmail.com](mailto:nik.j...@gmail.com) a écrit : > > > Hey > > Thanks much for the reply. Could you please give me a quick opinion on the jdbc river plugin? is it stable ? > > > Thanks > > > On Friday, May 17, 2013 1:13:59 AM UTC+5:30, Clinton Gormley wrote: > > Hi Nik > > > I don't recommend using the mongodb river. It is buggy and incomplete and not supported by Elasticsearch. > > > Instead, I'd just push your documents into Elasticsearch using the bulk API:[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/bulk/) > > > You'll save yourself a lot of time 🙂 > > > clint > > > On 16 May 2013 20:47, [nik....@gmail.com](mailto:nik....@gmail.com) wrote: > > OK I executed : > > curl -XPOST 'localhost:9200/\_river/\_open' > > > and then reran the command which gave the following error: > > {"error":"UnavailableShardsException[[\_river][0] [2] shardIt, [0] active : Timeout waiting for [1m], request: index {[\_river][some\_db][\_meta], source[\n{\n "type": "mongodb",\n "mongodb": { > > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > For more options, visithttps://groups.google.com/groups/opt\_out. > > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visithttps://groups.google.com/groups/opt\_out. > > smime.p7s > 1KViewDownload -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 17, 2013, 8:52am UTC](https://discuss.elastic.co/t/newbie-trying-to-setup-elastic-search-with-mongodb-and-nodejs/11996/8 "2013-05-17T08:52:33Z")
Using NodeJS? I don't know. Are you using any specific Client? In Java: [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/java-api/bulk/) Using CURL, the example here works: [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/bulk/) If you don't succeed in using it, please provide a full curl recreation of what you tried so far. HTH -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 17 mai 2013 à 10:39, Goddamnit [nik.jain@gmail.com](mailto:nik.jain@gmail.com) a écrit : > Thanks david, > Is there any solid working full example of using the bulk api ? THe > docs are just a reference and difficult for me to comprehend. I > understand of course that a simple curl request would deal with > insertion of data into the index. > > On May 17, 5:00 am, David Pilato [da...@pilato.fr](mailto:da...@pilato.fr) wrote: > > > Where are your source docs? > > In a SQL DB or in MongoDB or both? > > > > IMHO, JDBC river works fine but I prefer that Jörg answers directly to this question as he created the plugin. > > > > That said, you should be aware of that JDBC river is pulling docs every x minute. It's not a near real time indexing job. > > > > Let me say that in an other way. If you have a service layer in your application and from the service layer you persist your data in any SQL database, just send it at the same time to Elasticsearch for indexing. That way, you will have a finer control of what you are indexing and it will be done near real time… > > > > My 2 cents. > > > > -- > > David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) > > @dadoonet | @elasticsearchfr | @scrutmydocs > > > > Le 16 mai 2013 à 23:09, [nik.j...@gmail.com](mailto:nik.j...@gmail.com) a écrit : > > > > > Hey > > > Thanks much for the reply. Could you please give me a quick opinion on the jdbc river plugin? is it stable ? > > > > > Thanks > > > > > On Friday, May 17, 2013 1:13:59 AM UTC+5:30, Clinton Gormley wrote: > > > Hi Nik > > > > > I don't recommend using the mongodb river. It is buggy and incomplete and not supported by Elasticsearch. > > > > > Instead, I'd just push your documents into Elasticsearch using the bulk API:[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/bulk/) > > > > > You'll save yourself a lot of time 🙂 > > > > > clint > > > > > On 16 May 2013 20:47, [nik....@gmail.com](mailto:nik....@gmail.com) wrote: > > > OK I executed : > > > curl -XPOST 'localhost:9200/\_river/\_open' > > > > > and then reran the command which gave the following error: > > > {"error":"UnavailableShardsException[[\_river][0] [2] shardIt, [0] active : Timeout waiting for [1m], request: index {[\_river][some\_db][\_meta], source[\n{\n "type": "mongodb",\n "mongodb": { > > > > > -- > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visithttps://groups.google.com/groups/opt\_out. > > > > > -- > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visithttps://groups.google.com/groups/opt\_out. > > > > smime.p7s > > 1KViewDownload > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [May 17, 2013, 4:01pm UTC](https://discuss.elastic.co/t/newbie-trying-to-setup-elastic-search-with-mongodb-and-nodejs/11996/9 "2013-05-17T16:01:34Z")
If you are using the Java client, then looking at the source of any of the existing indexing rivers will provide some insight on how bulk indexing works. The rivers are singleton applications that use the same APIs as any other client. The official Elasticsearch rivers should all be using the BulkProcessor class, which is probably easier to understand and more efficient. -- Ivan On Fri, May 17, 2013 at 1:39 AM, Goddamnit [nik.jain@gmail.com](mailto:nik.jain@gmail.com) wrote: > Thanks david, > Is there any solid working full example of using the bulk api ? THe > docs are just a reference and difficult for me to comprehend. I > understand of course that a simple curl request would deal with > insertion of data into the index. > > On May 17, 5:00 am, David Pilato [da...@pilato.fr](mailto:da...@pilato.fr) wrote: > > > Where are your source docs? > > In a SQL DB or in MongoDB or both? > > > > IMHO, JDBC river works fine but I prefer that Jörg answers directly to > > this question as he created the plugin. > > > > That said, you should be aware of that JDBC river is pulling docs every > > x minute. It's not a near real time indexing job. > > > > Let me say that in an other way. If you have a service layer in your > > application and from the service layer you persist your data in any SQL > > database, just send it at the same time to Elasticsearch for indexing. That > > way, you will have a finer control of what you are indexing and it will be > > done near real time… > > > > My 2 cents. > > > > -- > > David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) > > @dadoonet | @elasticsearchfr | @scrutmydocs > > > > Le 16 mai 2013 à 23:09, [nik.j...@gmail.com](mailto:nik.j...@gmail.com) a écrit : > > > > > Hey > > > Thanks much for the reply. Could you please give me a quick opinion on > > > the jdbc river plugin? is it stable ? > > > > > Thanks > > > > > On Friday, May 17, 2013 1:13:59 AM UTC+5:30, Clinton Gormley wrote: > > > Hi Nik > > > > > I don't recommend using the mongodb river. It is buggy and incomplete > > > and not supported by Elasticsearch. > > > > > Instead, I'd just push your documents into Elasticsearch using the > > > bulk API:[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/bulk/) > > > > > You'll save yourself a lot of time 🙂 > > > > > clint > > > > > On 16 May 2013 20:47, [nik....@gmail.com](mailto:nik....@gmail.com) wrote: > > > OK I executed : > > > curl -XPOST 'localhost:9200/\_river/\_open' > > > > > and then reran the command which gave the following error: > > > {"error":"UnavailableShardsException[[\_river][0] [2] shardIt, [0] > > > active : Timeout waiting for [1m], request: index > > > {[\_river][some\_db][\_meta], source[\n{\n "type": "mongodb",\n > > > "mongodb": { > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visithttps://groups.google.com/groups/opt\_out. > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visithttps://groups.google.com/groups/opt\_out. > > > > smime.p7s > > 1KViewDownload > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:36am UTC](https://discuss.elastic.co/t/newbie-trying-to-setup-elastic-search-with-mongodb-and-nodejs/11996/10 "2017-07-06T02:36:01Z")