# Slow to index **URL:** \ **Category:** Elasticsearch\ **Created:** [August 24, 2011, 2:46pm UTC](https://discuss.elastic.co/t/slow-to-index/5238 "2011-08-24T14:46:25Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Harry\_Waye](https://avatars.discourse-cdn.com/v4/letter/h/f04885/32.png) [@Harry\_Waye](https://discuss.elastic.co/u/Harry_Waye)\ **Post date:** [August 24, 2011, 2:46pm UTC](https://discuss.elastic.co/t/slow-to-index/5238/1 "2011-08-24T14:46:25Z")
Hi, I'm on elasticsearch master with two nodes, one index or 5 shards and 1 replica. I'm rivering in a couchdb database of about 3GB of 400000 documents. It's going very slowly, something like 50 documents a minute. I can see on the rivering node that CPU usage is maxed out so its working pretty hard. The river and index are defined as per [https://gist.github.com/1154061](https://gist.github.com/1154061) and node config is [https://gist.github.com/1156885](https://gist.github.com/1156885) . How would I go about debugging this? Regards, Harry ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 24, 2011, 5:37pm UTC](https://discuss.elastic.co/t/slow-to-index/5238/2 "2011-08-24T17:37:55Z")
Try maybe increasing the bulk timeout? So more documents will get into a single bulk request. How complex are your docs? On Wed, Aug 24, 2011 at 5:46 PM, Harry Waye [hwaye@microwayes.net](mailto:hwaye@microwayes.net) wrote: > Hi, I'm on elasticsearch master with two nodes, one index or 5 shards and 1 > replica. I'm rivering in a couchdb database of about 3GB of 400000 > documents. It's going very slowly, something like 50 documents a minute. I > can see on the rivering node that CPU usage is maxed out so its working > pretty hard. > > The river and index are defined as per [gist:1154061 路 GitHub](https://gist.github.com/1154061) > and node config is [gist:1156885 路 GitHub](https://gist.github.com/1156885) . > > How would I go about debugging this? > > Regards, > Harry ---
**Author:** ![Harry\_Waye](https://avatars.discourse-cdn.com/v4/letter/h/f04885/32.png) [@Harry\_Waye](https://discuss.elastic.co/u/Harry_Waye)\ **Post date:** [August 25, 2011, 9:27am UTC](https://discuss.elastic.co/t/slow-to-index/5238/3 "2011-08-25T09:27:55Z")
I just tried increasing to 1s to no avail. The documents vary from about 1k to 150k, with 100 distinct attributes overall, but each document only including a small number of these, about 10. I'm going to try a 30s timeout now to see it that makes a difference, and will play around with the bulk\_size. Are there any other settings that I can twiddle? ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [August 25, 2011, 10:18am UTC](https://discuss.elastic.co/t/slow-to-index/5238/4 "2011-08-25T10:18:29Z")
On Thu, 2011-08-25 at 02:27 -0700, Harry Waye wrote: > I just tried increasing to 1s to no avail. The documents vary from > about 1k to 150k, with 100 distinct attributes overall, but each > document only including a small number of these, about 10. I'm going > to try a 30s timeout now to see it that makes a difference, and will > play around with the bulk\_size. Are there any other settings that I > can twiddle? How much memory have you allocated to ES? And have you made sure that swap is disabled, either by turning swapoff completely, or by using mlockall? Swap is your enemy - as soon as any part of the heap is in swap, the JVM will grind to a halt. clint ---
**Author:** ![Harry\_Waye](https://avatars.discourse-cdn.com/v4/letter/h/f04885/32.png) [@Harry\_Waye](https://discuss.elastic.co/u/Harry_Waye)\ **Post date:** [August 25, 2011, 10:27am UTC](https://discuss.elastic.co/t/slow-to-index/5238/5 "2011-08-25T10:27:15Z")
Min/max is set to 500/1000. I hadn't turned swap off so will give that a try now... ---
**Author:** ![Harry\_Waye](https://avatars.discourse-cdn.com/v4/letter/h/f04885/32.png) [@Harry\_Waye](https://discuss.elastic.co/u/Harry_Waye)\ **Post date:** [August 25, 2011, 10:40am UTC](https://discuss.elastic.co/t/slow-to-index/5238/6 "2011-08-25T10:40:17Z")
No change, still very slow ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 25, 2011, 2:29pm UTC](https://discuss.elastic.co/t/slow-to-index/5238/7 "2011-08-25T14:29:28Z")
Wondering, do you see heavy CPU load also on the other node? On Thu, Aug 25, 2011 at 1:40 PM, Harry Waye [hwaye@microwayes.net](mailto:hwaye@microwayes.net) wrote: > No change, still very slow ---
**Author:** ![Harry\_Waye\_2](https://avatars.discourse-cdn.com/v4/letter/h/b2d939/32.png) [@Harry\_Waye\_2](https://discuss.elastic.co/u/Harry_Waye_2)\ **Post date:** [August 25, 2011, 2:35pm UTC](https://discuss.elastic.co/t/slow-to-index/5238/8 "2011-08-25T14:35:00Z")
No, just on the rivering node as I recall. I've disbanded the the group so can't verify easily. I'll have to arrange a reunion later to test. On 25 August 2011 15:29, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > Wondering, do you see heavy CPU load also on the other node? > > On Thu, Aug 25, 2011 at 1:40 PM, Harry Waye [hwaye@microwayes.net](mailto:hwaye@microwayes.net) wrote: > > > No change, still very slow ---
**Author:** ![Harry\_Waye\_2](https://avatars.discourse-cdn.com/v4/letter/h/b2d939/32.png) [@Harry\_Waye\_2](https://discuss.elastic.co/u/Harry_Waye_2)\ **Post date:** [August 25, 2011, 3:01pm UTC](https://discuss.elastic.co/t/slow-to-index/5238/9 "2011-08-25T15:01:36Z")
We've noticed that the river is pulling in \_attachments as well, is it meant to be doing that? On 25 August 2011 15:35, Harry Waye [harry@arachnys.com](mailto:harry@arachnys.com) wrote: > No, just on the rivering node as I recall. I've disbanded the the group so > can't verify easily. I'll have to arrange a reunion later to test. > > On 25 August 2011 15:29, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > Wondering, do you see heavy CPU load also on the other node? > > > > On Thu, Aug 25, 2011 at 1:40 PM, Harry Waye [hwaye@microwayes.net](mailto:hwaye@microwayes.net) wrote: > > > > > No change, still very slow ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [August 25, 2011, 3:17pm UTC](https://discuss.elastic.co/t/slow-to-index/5238/10 "2011-08-25T15:17:23Z")
Yes. I will submit a pull request to disable it with a parameter next week. By now, you have to add a script like ctx.\_doc.attachement=null Btw, you will have to add the javascript plugin. Hope this helps David 馃槈 Le 25 ao没t 2011 脿 17:01, Harry Waye [harry@arachnys.com](mailto:harry@arachnys.com) a 茅crit : > We've noticed that the river is pulling in \_attachments as well, is it meant to be doing that? > > On 25 August 2011 15:35, Harry Waye [harry@arachnys.com](mailto:harry@arachnys.com) wrote: > No, just on the rivering node as I recall. I've disbanded the the group so can't verify easily. I'll have to arrange a reunion later to test. > > On 25 August 2011 15:29, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > Wondering, do you see heavy CPU load also on the other node? > > On Thu, Aug 25, 2011 at 1:40 PM, Harry Waye [hwaye@microwayes.net](mailto:hwaye@microwayes.net) wrote: > No change, still very slow ---
**Author:** ![Harry\_Waye\_2](https://avatars.discourse-cdn.com/v4/letter/h/b2d939/32.png) [@Harry\_Waye\_2](https://discuss.elastic.co/u/Harry_Waye_2)\ **Post date:** [August 25, 2011, 3:52pm UTC](https://discuss.elastic.co/t/slow-to-index/5238/11 "2011-08-25T15:52:59Z")
Thanks David On 25 August 2011 16:17, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote: > Yes. > I will submit a pull request to disable it with a parameter next week. > By now, you have to add a script like ctx.\_doc.attachement=null > > Btw, you will have to add the javascript plugin. > > Hope this helps > David 馃槈 > > Le 25 ao没t 2011 脿 17:01, Harry Waye [harry@arachnys.com](mailto:harry@arachnys.com) a 茅crit : > > We've noticed that the river is pulling in \_attachments as well, is it > meant to be doing that? > > On 25 August 2011 15:35, Harry Waye \< [harry@arachnys.com](mailto:harry@arachnys.com) > [harry@arachnys.com](mailto:harry@arachnys.com)\> wrote: > > > No, just on the rivering node as I recall. I've disbanded the the group > > so can't verify easily. I'll have to arrange a reunion later to test. > > > > On 25 August 2011 15:29, Shay Banon \< [kimchy@gmail.com](mailto:kimchy@gmail.com)[kimchy@gmail.com](mailto:kimchy@gmail.com)\>wrote: > > > > > Wondering, do you see heavy CPU load also on the other node? > > > > > > On Thu, Aug 25, 2011 at 1:40 PM, Harry Waye \< [hwaye@microwayes.net](mailto:hwaye@microwayes.net) > > > [hwaye@microwayes.net](mailto:hwaye@microwayes.net)\> wrote: > > > > > > > No change, still very slow ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 26, 2011, 2:10pm UTC](https://discuss.elastic.co/t/slow-to-index/5238/12 "2011-08-26T14:10:04Z")
What does that mean, that it pulls attachments? Can we disable pulling attachments on the \_changes stream itself if one does not wish to have them? On Thu, Aug 25, 2011 at 6:17 PM, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote: > Yes. > I will submit a pull request to disable it with a parameter next week. > By now, you have to add a script like ctx.\_doc.attachement=null > > Btw, you will have to add the javascript plugin. > > Hope this helps > David 馃槈 > > Le 25 ao没t 2011 脿 17:01, Harry Waye [harry@arachnys.com](mailto:harry@arachnys.com) a 茅crit : > > We've noticed that the river is pulling in \_attachments as well, is it > meant to be doing that? > > On 25 August 2011 15:35, Harry Waye \< [harry@arachnys.com](mailto:harry@arachnys.com) > [harry@arachnys.com](mailto:harry@arachnys.com)\> wrote: > > > No, just on the rivering node as I recall. I've disbanded the the group > > so can't verify easily. I'll have to arrange a reunion later to test. > > > > On 25 August 2011 15:29, Shay Banon \< [kimchy@gmail.com](mailto:kimchy@gmail.com)[kimchy@gmail.com](mailto:kimchy@gmail.com)\>wrote: > > > > > Wondering, do you see heavy CPU load also on the other node? > > > > > > On Thu, Aug 25, 2011 at 1:40 PM, Harry Waye \< [hwaye@microwayes.net](mailto:hwaye@microwayes.net) > > > [hwaye@microwayes.net](mailto:hwaye@microwayes.net)\> wrote: > > > > > > > No change, still very slow ---
**Author:** ![Harry\_Waye\_2](https://avatars.discourse-cdn.com/v4/letter/h/b2d939/32.png) [@Harry\_Waye\_2](https://discuss.elastic.co/u/Harry_Waye_2)\ **Post date:** [August 26, 2011, 2:22pm UTC](https://discuss.elastic.co/t/slow-to-index/5238/13 "2011-08-26T14:22:05Z")
Not that it pulls in attachments, just attachment metadata list content\_type, length etc. Each attachment is assigned a hash so you end up with many many fields, several for each attachment. I don't think you can suppress the field, perhaps theres some was of using a view but we're just removing it elasticsearch side for now. On 26 August 2011 15:10, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > What does that mean, that it pulls attachments? Can we disable pulling > attachments on the \_changes stream itself if one does not wish to have them? > > On Thu, Aug 25, 2011 at 6:17 PM, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote: > > > Yes. > > I will submit a pull request to disable it with a parameter next week. > > By now, you have to add a script like ctx.\_doc.attachement=null > > > > Btw, you will have to add the javascript plugin. > > > > Hope this helps > > David 馃槈 > > > > Le 25 ao没t 2011 脿 17:01, Harry Waye [harry@arachnys.com](mailto:harry@arachnys.com) a 茅crit : > > > > We've noticed that the river is pulling in \_attachments as well, is it > > meant to be doing that? > > > > On 25 August 2011 15:35, Harry Waye \< [harry@arachnys.com](mailto:harry@arachnys.com) > > [harry@arachnys.com](mailto:harry@arachnys.com)\> wrote: > > > > > No, just on the rivering node as I recall. I've disbanded the the group > > > so can't verify easily. I'll have to arrange a reunion later to test. > > > > > > On 25 August 2011 15:29, Shay Banon \< [kimchy@gmail.com](mailto:kimchy@gmail.com)[kimchy@gmail.com](mailto:kimchy@gmail.com) > > > > > > > wrote: > > > > > > > Wondering, do you see heavy CPU load also on the other node? > > > > > > > > On Thu, Aug 25, 2011 at 1:40 PM, Harry Waye \< [hwaye@microwayes.net](mailto:hwaye@microwayes.net) > > > > [hwaye@microwayes.net](mailto:hwaye@microwayes.net)\> wrote: > > > > > > > > > No change, still very slow ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:55am UTC](https://discuss.elastic.co/t/slow-to-index/5238/14 "2017-07-06T03:55:56Z")