# How to control on ES nodes while doing bulk indexing through river **URL:** \ **Category:** Elasticsearch\ **Created:** [July 9, 2013, 10:45pm UTC](https://discuss.elastic.co/t/how-to-control-on-es-nodes-while-doing-bulk-indexing-through-river/12730 "2013-07-09T22:45:34Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![MJR\_M](https://avatars.discourse-cdn.com/v4/letter/m/8c91f0/32.png) [@MJR\_M](https://discuss.elastic.co/u/MJR_M)\ **Post date:** [July 9, 2013, 10:45pm UTC](https://discuss.elastic.co/t/how-to-control-on-es-nodes-while-doing-bulk-indexing-through-river/12730/1 "2013-07-09T22:45:34Z")
Hi ``` In our prod ES cluster, we have 5 nodes and I have to re-index all the ``` documents. I am doing bulk indexing through river, and while doing this load on ES nodes increase to 10 - 15+, and obviously this slows down the search. Can anyone suggest what should I do in order to bulk index through river and at the same time keep the load on nodes under control. I have to re-index 50 million documents. Regards MJR -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [July 10, 2013, 8:50am UTC](https://discuss.elastic.co/t/how-to-control-on-es-nodes-while-doing-bulk-indexing-through-river/12730/2 "2013-07-10T08:50:30Z")
Hey, you can decrease the bulk thread pool size, you can change or disable the refresh interval, you can enable merge throttling (which enabled by default on newer versions). As you havent provided any information about the elasticsearch version you are using or the configuration you have applied, it is hard to go into any detail, but the above configuration options might be a start. --Alex On Wed, Jul 10, 2013 at 12:45 AM, MJR M [emjayaarr@gmail.com](mailto:emjayaarr@gmail.com) wrote: > Hi > > ``` > In our prod ES cluster, we have 5 nodes and I have to re-index all > > ``` > > the documents. I am doing bulk indexing through river, and while doing this > load on ES nodes increase to 10 - 15+, and obviously this slows down the > search. Can anyone suggest what should I do in order to bulk index through > river and at the same time keep the load on nodes under control. I have to > re-index 50 million documents. > > Regards > MJR > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [July 10, 2013, 12:42pm UTC](https://discuss.elastic.co/t/how-to-control-on-es-nodes-while-doing-bulk-indexing-through-river/12730/3 "2013-07-10T12:42:32Z")
if you want both - max perf on load and max perf on search - you should use two indexes, one for the old generation and one for new generation, und connect them with an index alias. Distribute the indexes over the nodes so they form two separated groups, that is, so they use different machines (for example, by shard moving, shard allocation). Set replica level to 0 (no replicas) for the new gen index. Forward search only to those nodes with the old gen. After bulk is complete, add replica level to new gen, and switch from old to new with the help of index alias (or by just dropping the old gen). You may see a perf hit when replicas are building up but this is not much compared to bulk load. Jörg Am 10.07.13 00:45, schrieb MJR M: > Hi > > ``` > In our prod ES cluster, we have 5 nodes and I have to re-index > > ``` > > all the documents. I am doing bulk indexing through river, and while > doing this load on ES nodes increase to 10 - 15+, and obviously this > slows down the search. Can anyone suggest what should I do in order to > bulk index through river and at the same time keep the load on nodes > under control. I have to re-index 50 million documents. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![MJR\_M](https://avatars.discourse-cdn.com/v4/letter/m/8c91f0/32.png) [@MJR\_M](https://discuss.elastic.co/u/MJR_M)\ **Post date:** [July 10, 2013, 8:11pm UTC](https://discuss.elastic.co/t/how-to-control-on-es-nodes-while-doing-bulk-indexing-through-river/12730/4 "2013-07-10T20:11:42Z")
Thanks Alex, will try your suggestions. We are using ES 0.20.4, not upgraded to 0.90.2 yet. On Wednesday, July 10, 2013 1:50:30 AM UTC-7, Alexander Reelsen wrote: > Hey, > > you can decrease the bulk thread pool size, you can change or disable the > refresh interval, you can enable merge throttling (which enabled by default > on newer versions). As you havent provided any information about the > elasticsearch version you are using or the configuration you have applied, > it is hard to go into any detail, but the above configuration options might > be a start. > > --Alex > > On Wed, Jul 10, 2013 at 12:45 AM, MJR M \<[emja...@gmail.com](mailto:emja...@gmail.com) \\>wrote: > > > Hi > > > > ``` > > In our prod ES cluster, we have 5 nodes and I have to re-index all > > > > ``` > > > > the documents. I am doing bulk indexing through river, and while doing this > > load on ES nodes increase to 10 - 15+, and obviously this slows down the > > search. Can anyone suggest what should I do in order to bulk index through > > river and at the same time keep the load on nodes under control. I have to > > re-index 50 million documents. > > > > Regards > > MJR > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![MJR\_M](https://avatars.discourse-cdn.com/v4/letter/m/8c91f0/32.png) [@MJR\_M](https://discuss.elastic.co/u/MJR_M)\ **Post date:** [July 10, 2013, 8:12pm UTC](https://discuss.elastic.co/t/how-to-control-on-es-nodes-while-doing-bulk-indexing-through-river/12730/5 "2013-07-10T20:12:09Z")
Thanks Jorg for your suggestions. On Wednesday, July 10, 2013 5:42:32 AM UTC-7, Jörg Prante wrote: > if you want both - max perf on load and max perf on search - you should > use two indexes, one for the old generation and one for new generation, > und connect them with an index alias. > Distribute the indexes over the nodes so they form two separated groups, > that is, so they use different machines (for example, by shard moving, > shard allocation). Set replica level to 0 (no replicas) for the new gen > index. Forward search only to those nodes with the old gen. After bulk > is complete, add replica level to new gen, and switch from old to new > with the help of index alias (or by just dropping the old gen). You may > see a perf hit when replicas are building up but this is not much > compared to bulk load. > > Jörg > > Am 10.07.13 00:45, schrieb MJR M: > > > Hi > > > > ``` > > In our prod ES cluster, we have 5 nodes and I have to re-index > > > > ``` > > > > all the documents. I am doing bulk indexing through river, and while > > doing this load on ES nodes increase to 10 - 15+, and obviously this > > slows down the search. Can anyone suggest what should I do in order to > > bulk index through river and at the same time keep the load on nodes > > under control. I have to re-index 50 million documents. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:27am UTC](https://discuss.elastic.co/t/how-to-control-on-es-nodes-while-doing-bulk-indexing-through-river/12730/6 "2017-07-06T02:27:05Z")