# How to Speed Up Indexing **URL:** \ **Category:** Elasticsearch\ **Created:** [September 2, 2014, 4:46am UTC](https://discuss.elastic.co/t/how-to-speed-up-indexing/19565 "2014-09-02T04:46:19Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![xiehaiwei](https://avatars.discourse-cdn.com/v4/letter/x/6de8d8/32.png) [@xiehaiwei](https://discuss.elastic.co/u/xiehaiwei)\ **Post date:** [September 2, 2014, 4:46am UTC](https://discuss.elastic.co/t/how-to-speed-up-indexing/19565/1 "2014-09-02T04:46:19Z")
Hi all, In our ES system, one line of a Mysql table will be indexing as a document, but indexing speed is slow. My Questions: 1. how fast of using BulkAPI indexing compared with single indexing? 2. If ’Word Segmentation‘ is the problem, how to deal it? 3. Can I use multi nodes of ES cluster to parallelly indexing in one Index? Thanks. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4f7eae49-1bee-4bdd-9a8c-c9d1178fccdc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4f7eae49-1bee-4bdd-9a8c-c9d1178fccdc%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![vineeth\_mohan\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vineeth_mohan_2/32/747_2.png) [@vineeth\_mohan\_2](https://discuss.elastic.co/u/vineeth_mohan_2)\ **Post date:** [September 2, 2014, 5:41am UTC](https://discuss.elastic.co/t/how-to-speed-up-indexing/19565/2 "2014-09-02T05:41:20Z")
Hello , One tip from my experience - 1. Disable refresh before bulk indexing and enable it once its done. ES waits for 1 second and then make all documents which are indexed during that time , searchable. - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-update-settings.html#bulk) 2. Reduce replica to 0 while bulk indexing. 3. Increase number of machines and add the shard number . The indexing is happening in parallel. So more machines with a shard in it will help. "If ’Word Segmentation‘ is the problem" - Please elaborate. Thanks Vineeth On Tue, Sep 2, 2014 at 10:16 AM, [xiehaiwei@gmail.com](mailto:xiehaiwei@gmail.com) wrote: > Hi all, > In our ES system, one line of a Mysql table will be indexing as a > document, but indexing speed is slow. > > My Questions: > > 1. how fast of using BulkAPI indexing compared with single indexing? > 2. If ’Word Segmentation‘ is the problem, how to deal it? > 3. Can I use multi nodes of ES cluster to parallelly indexing in one > Index? > > Thanks. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/4f7eae49-1bee-4bdd-9a8c-c9d1178fccdc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4f7eae49-1bee-4bdd-9a8c-c9d1178fccdc%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/4f7eae49-1bee-4bdd-9a8c-c9d1178fccdc%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4f7eae49-1bee-4bdd-9a8c-c9d1178fccdc%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGdPd5kP\_adjHNyMoC5-VTzt6%2ByX8bEhfWmH3KFaCtDYiSQ8Mg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGdPd5kP_adjHNyMoC5-VTzt6%2ByX8bEhfWmH3KFaCtDYiSQ8Mg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![xiehaiwei](https://avatars.discourse-cdn.com/v4/letter/x/6de8d8/32.png) [@xiehaiwei](https://discuss.elastic.co/u/xiehaiwei)\ **Post date:** [September 2, 2014, 6:43am UTC](https://discuss.elastic.co/t/how-to-speed-up-indexing/19565/3 "2014-09-02T06:43:08Z")
Hi, ``` "If ’Word Segmentation‘ is the problem" - means, word ``` segmentation analyzer speed is not good, about 1MB/s when runs independently. In our case, many fields of a document need to be segment. ``` "more machines with a shard" - Will a shard be running in multi ``` nodes? Do you mean with a cluster? Thanks. Haiwei > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2d601d98-18c9-4e63-bffc-6948a072e30a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/2d601d98-18c9-4e63-bffc-6948a072e30a%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![vineeth\_mohan\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vineeth_mohan_2/32/747_2.png) [@vineeth\_mohan\_2](https://discuss.elastic.co/u/vineeth_mohan_2)\ **Post date:** [September 2, 2014, 8:52am UTC](https://discuss.elastic.co/t/how-to-speed-up-indexing/19565/4 "2014-09-02T08:52:52Z")
Hello Haiwei , The more hardware you can get , it should be better unless the data is too small. So if there are 10 machines , set the shards as 10 , so that the index can uniformly use all the resources. Thanks Vineeth On Tue, Sep 2, 2014 at 12:13 PM, [xiehaiwei@gmail.com](mailto:xiehaiwei@gmail.com) wrote: > Hi, > > ``` > "If ’Word Segmentation‘ is the problem" - means, word > > ``` > > segmentation analyzer speed is not good, > about 1MB/s when runs independently. In our case, many fields of a > document need to be segment. > > ``` > "more machines with a shard" - Will a shard be running in multi > > ``` > > nodes? Do you mean with a cluster? > > Thanks. > Haiwei > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/2d601d98-18c9-4e63-bffc-6948a072e30a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/2d601d98-18c9-4e63-bffc-6948a072e30a%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/2d601d98-18c9-4e63-bffc-6948a072e30a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2d601d98-18c9-4e63-bffc-6948a072e30a%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGdPd5m55uabMdvycqd-VO8jwxU1pcMnjh1QqYO1W\_cc4ss9\_w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGdPd5m55uabMdvycqd-VO8jwxU1pcMnjh1QqYO1W_cc4ss9_w%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![xiehaiwei](https://avatars.discourse-cdn.com/v4/letter/x/6de8d8/32.png) [@xiehaiwei](https://discuss.elastic.co/u/xiehaiwei)\ **Post date:** [September 2, 2014, 9:05am UTC](https://discuss.elastic.co/t/how-to-speed-up-indexing/19565/5 "2014-09-02T09:05:07Z")
Hi, mohan ``` My lastest testing, indixing data about 14000 documents. ``` 1. Tuning BulkAPI params, 6m. Before tuning, time is 14m. [INFO] Total time: 6:06.173s [INFO] Finished at: Tue Sep 02 15:40:36 CST 2014 [INFO] Final Memory: 27M/312M ref: [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-update-settings.html#bulk](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-update-settings.html#bulk) 2. Close Analyzer of Strings, 18s. [INFO] Total time: 18.499s [INFO] Finished at: Tue Sep 02 15:52:47 CST 2014 [INFO] Final Memory: 29M/312M So, Is Analyzer of Strings indexing the bigest problem of perfomance? Thanks. Haiwei. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/332f63cd-84f0-494a-9868-5ac9a702f2b2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/332f63cd-84f0-494a-9868-5ac9a702f2b2%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:05am UTC](https://discuss.elastic.co/t/how-to-speed-up-indexing/19565/6 "2017-07-06T01:05:00Z")