# CouchDB River and memory **URL:** \ **Category:** Elasticsearch\ **Created:** [August 19, 2011, 2:34pm UTC](https://discuss.elastic.co/t/couchdb-river-and-memory/5200 "2011-08-19T14:34:45Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Harry\_Waye](https://avatars.discourse-cdn.com/v4/letter/h/f04885/32.png) [@Harry\_Waye](https://discuss.elastic.co/u/Harry_Waye)\ **Post date:** [August 19, 2011, 2:34pm UTC](https://discuss.elastic.co/t/couchdb-river-and-memory/5200/1 "2011-08-19T14:34:45Z")
I'm trying to index a 1.5GB couchdb database with the CouchDB river but I'm getting memory issues. elasticsearch.conf is > The river and index were created with [https://gist.github.com/1156889](https://gist.github.com/1156889) . The node on which the river is running has 1.4GB of memory allocated to the JVM. I end up running in to OOM errors after a couple of minutes: [https://gist.github.com/1156931](https://gist.github.com/1156931) . I'm also getting mapper exceptions on interpreting dates but I assume these are a separate issue. What should I be expecting the memory usage to look like? Thanks Harry ---
**Author:** ![avasilenko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/avasilenko/32/2901_2.png) [@avasilenko](https://discuss.elastic.co/u/avasilenko)\ **Post date:** [August 22, 2011, 7:02am UTC](https://discuss.elastic.co/t/couchdb-river-and-memory/5200/2 "2011-08-22T07:02:19Z")
> I'm also getting mapper exceptions on interpreting dates but I assume > these are a separate issue. What ES version are you using? There was a hotfix in 0.17.6 - > **[Issues · elastic/elasticsearch](https://github.com/elastic/elasticsearch/issues?q=is%3Aclosed%2Bsort%3Acreated-desc%2Blabel%3Av0.17.6)** > > Free and Open, Distributed, RESTful Search Engine. Contribute to elastic/elasticsearch development by creating an account on GitHub. Regards, Alexandr Vasilenko 2011/8/19 Harry Waye [hwaye@microwayes.net](mailto:hwaye@microwayes.net) > I'm trying to index a 1.5GB couchdb database with the CouchDB river but I'm > getting memory issues. elasticsearch.conf is > [gist:1156885 · GitHub](https://gist.github.com/1156885) > > The river and index were created with [gist:1156889 · GitHub](https://gist.github.com/1156889) . > The node on which the river is running has 1.4GB of memory allocated to the > JVM. I end up running in to OOM errors after a couple of minutes: > [gist:1156931 · GitHub](https://gist.github.com/1156931) . I'm also getting mapper exceptions on > interpreting dates but I assume these are a separate issue. What should I > be expecting the memory usage to look like? > > Thanks > Harry ---
**Author:** ![Harry\_Waye\_2](https://avatars.discourse-cdn.com/v4/letter/h/b2d939/32.png) [@Harry\_Waye\_2](https://discuss.elastic.co/u/Harry_Waye_2)\ **Post date:** [August 22, 2011, 9:30am UTC](https://discuss.elastic.co/t/couchdb-river-and-memory/5200/3 "2011-08-22T09:30:21Z")
This is in 0.17.6. Is the date issue likely to be related to the OOM exception? Let me know if there is anything I can do, more information etc. Regards Harry On 22 August 2011 08:02, Alex Vasilenko [aa.vasilenko@gmail.com](mailto:aa.vasilenko@gmail.com) wrote: > I'm also getting mapper exceptions on interpreting dates but I assume > > > these are a separate issue. > > What ES version are you using? There was a hotfix in 0.17.6 - > [Issues · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues?labels=v0.17.6&sort=created&direction=desc&state=closed&page=1) > > Regards, > Alexandr Vasilenko > > 2011/8/19 Harry Waye [hwaye@microwayes.net](mailto:hwaye@microwayes.net) > > > I'm trying to index a 1.5GB couchdb database with the CouchDB river but > > I'm getting memory issues. elasticsearch.conf is > > [gist:1156885 · GitHub](https://gist.github.com/1156885) > > > > The river and index were created with [gist:1156889 · GitHub](https://gist.github.com/1156889) . > > The node on which the river is running has 1.4GB of memory allocated to the > > JVM. I end up running in to OOM errors after a couple of minutes: > > [gist:1156931 · GitHub](https://gist.github.com/1156931) . I'm also getting mapper exceptions on > > interpreting dates but I assume these are a separate issue. What should I > > be expecting the memory usage to look like? > > > > Thanks > > Harry ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 22, 2011, 7:01pm UTC](https://discuss.elastic.co/t/couchdb-river-and-memory/5200/4 "2011-08-22T19:01:59Z")
How big are the docs you are trying to index? I wonder if its not indexing fast enough as its pulling data from couchdb (and we probably need a way to throttle if that happens). How many nodes are you running? On Mon, Aug 22, 2011 at 12:30 PM, Harry Waye [harry@arachnys.com](mailto:harry@arachnys.com) wrote: > This is in 0.17.6. Is the date issue likely to be related to the OOM > exception? Let me know if there is anything I can do, more information etc. > > Regards > Harry > > On 22 August 2011 08:02, Alex Vasilenko [aa.vasilenko@gmail.com](mailto:aa.vasilenko@gmail.com) wrote: > > > I'm also getting mapper exceptions on interpreting dates but I assume > > > > > these are a separate issue. > > > > What ES version are you using? There was a hotfix in 0.17.6 - > > [Issues · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues?labels=v0.17.6&sort=created&direction=desc&state=closed&page=1) > > > > Regards, > > Alexandr Vasilenko > > > > 2011/8/19 Harry Waye [hwaye@microwayes.net](mailto:hwaye@microwayes.net) > > > > > I'm trying to index a 1.5GB couchdb database with the CouchDB river but > > > I'm getting memory issues. elasticsearch.conf is > > > [gist:1156885 · GitHub](https://gist.github.com/1156885) > > > > > > The river and index were created with [gist:1156889 · GitHub](https://gist.github.com/1156889) . > > > The node on which the river is running has 1.4GB of memory allocated to the > > > JVM. I end up running in to OOM errors after a couple of minutes: > > > [gist:1156931 · GitHub](https://gist.github.com/1156931) . I'm also getting mapper exceptions on > > > interpreting dates but I assume these are a separate issue. What should I > > > be expecting the memory usage to look like? > > > > > > Thanks > > > Harry ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 22, 2011, 7:19pm UTC](https://discuss.elastic.co/t/couchdb-river-and-memory/5200/5 "2011-08-22T19:19:51Z")
Opened this issue: [CouchDB River: Add throttling when indexing does not keep up with fetching \_changes · Issue #1269 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1269). Should provide much better behavior of the couchdb river (its on 0.17 branch as well). On Mon, Aug 22, 2011 at 10:01 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > How big are the docs you are trying to index? I wonder if its not indexing > fast enough as its pulling data from couchdb (and we probably need a way to > throttle if that happens). How many nodes are you running? > > On Mon, Aug 22, 2011 at 12:30 PM, Harry Waye [harry@arachnys.com](mailto:harry@arachnys.com) wrote: > > > This is in 0.17.6. Is the date issue likely to be related to the OOM > > exception? Let me know if there is anything I can do, more information etc. > > > > Regards > > Harry > > > > On 22 August 2011 08:02, Alex Vasilenko [aa.vasilenko@gmail.com](mailto:aa.vasilenko@gmail.com) wrote: > > > > > I'm also getting mapper exceptions on interpreting dates but I assume > > > > > > > these are a separate issue. > > > > > > What ES version are you using? There was a hotfix in 0.17.6 - > > > [Issues · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues?labels=v0.17.6&sort=created&direction=desc&state=closed&page=1) > > > > > > Regards, > > > Alexandr Vasilenko > > > > > > 2011/8/19 Harry Waye [hwaye@microwayes.net](mailto:hwaye@microwayes.net) > > > > > > > I'm trying to index a 1.5GB couchdb database with the CouchDB river but > > > > I'm getting memory issues. elasticsearch.conf is > > > > [gist:1156885 · GitHub](https://gist.github.com/1156885) > > > > > > > > The river and index were created with [gist:1156889 · GitHub](https://gist.github.com/1156889) . > > > > The node on which the river is running has 1.4GB of memory allocated to the > > > > JVM. I end up running in to OOM errors after a couple of minutes: > > > > [gist:1156931 · GitHub](https://gist.github.com/1156931) . I'm also getting mapper exceptions > > > > on interpreting dates but I assume these are a separate issue. What should > > > > I be expecting the memory usage to look like? > > > > > > > > Thanks > > > > Harry ---
**Author:** ![Harry\_Waye](https://avatars.discourse-cdn.com/v4/letter/h/f04885/32.png) [@Harry\_Waye](https://discuss.elastic.co/u/Harry_Waye)\ **Post date:** [August 22, 2011, 9:46pm UTC](https://discuss.elastic.co/t/couchdb-river-and-memory/5200/6 "2011-08-22T21:46:22Z")
I suspect the index rate is probably the issue. I was originally running 3 ec2 large 32bit nodes, as per the original post. I've subsequently just written my own "river" which I presume due to its poor efficiency is working fine. I'll have to get back to you on the document size... H On 22 Aug 2011 20:20, "Shay Banon" [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: ---
**Author:** ![Harry\_Waye](https://avatars.discourse-cdn.com/v4/letter/h/f04885/32.png) [@Harry\_Waye](https://discuss.elastic.co/u/Harry_Waye)\ **Post date:** [August 22, 2011, 9:53pm UTC](https://discuss.elastic.co/t/couchdb-river-and-memory/5200/7 "2011-08-22T21:53:39Z")
Ah, thanks for the commit, I'll. Try this out tomorrow. H On 22 Aug 2011 22:46, "Harry Waye" [hwaye@microwayes.net](mailto:hwaye@microwayes.net) wrote: > I suspect the index rate is probably the issue. I was originally running 3 > ec2 large 32bit nodes, as per the original post. I've subsequently just > written my own "river" which I presume due to its poor efficiency is > working > fine. I'll have to get back to you on the document size... > > H > On 22 Aug 2011 20:20, "Shay Banon" [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: ---
**Author:** ![Harry\_Waye](https://avatars.discourse-cdn.com/v4/letter/h/f04885/32.png) [@Harry\_Waye](https://discuss.elastic.co/u/Harry_Waye)\ **Post date:** [August 23, 2011, 2:24pm UTC](https://discuss.elastic.co/t/couchdb-river-and-memory/5200/8 "2011-08-23T14:24:08Z")
Looks too be a success, cheers! On 22 August 2011 22:53, Harry Waye [hwaye@microwayes.net](mailto:hwaye@microwayes.net) wrote: > Ah, thanks for the commit, I'll. Try this out tomorrow. > > H > On 22 Aug 2011 22:46, "Harry Waye" [hwaye@microwayes.net](mailto:hwaye@microwayes.net) wrote: > > > I suspect the index rate is probably the issue. I was originally running > > 3 > > ec2 large 32bit nodes, as per the original post. I've subsequently just > > written my own "river" which I presume due to its poor efficiency is > > working > > fine. I'll have to get back to you on the document size... > > > > H > > On 22 Aug 2011 20:20, "Shay Banon" [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:56am UTC](https://discuss.elastic.co/t/couchdb-river-and-memory/5200/9 "2017-07-06T03:56:16Z")