# Pyes bulk insert problem **URL:** \ **Category:** Elasticsearch\ **Created:** [June 11, 2013, 6:28am UTC](https://discuss.elastic.co/t/pyes-bulk-insert-problem/12356 "2013-06-11T06:28:15Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Dmitry\_Babitsky](https://avatars.discourse-cdn.com/v4/letter/d/85e7bf/32.png) [@Dmitry\_Babitsky](https://discuss.elastic.co/u/Dmitry_Babitsky)\ **Post date:** [June 11, 2013, 6:28am UTC](https://discuss.elastic.co/t/pyes-bulk-insert-problem/12356/1 "2013-06-11T06:28:15Z")
Hi, I have a 200M documents index I would like to reindex. I wrote the following script that goes over documents in the old index and puts them with balk insert into the new index. The size of each bulk is 2000 documents. search\_obj = pyes.query.Search(query = pyes.query.MatchAllQuery(), start= resume\_from) old\_index\_iterator = self.esconn.search(search\_obj, self.index\_name) counter = 0 BULK\_SIZE = 2000 for doc in old\_index\_iterator: self.esconn._index_(doc=doc, doc\_type=DOC\_TYPE, index=new\_index\_name, id= doc.get\_id(),\* bulk=True\*) counter += 1 if counter % BULK\_SIZE == 0: self.logger.debug("Refreshing...") - ``` **self.esconn.refresh()** ``` - ``` self.logger.debug("Refresh done.") ``` self.esconn.refresh() Observation: 1. The speed that I get is very slow: around 150 documents / minute. 2. The time of the refresh operation is 0. 3. If I remove the index command (just read from the DB) - I speed up 10 times. Conclusion: The index ignores the \*bulk=True \*flag, and pushes every single document to the ES server. Anyone know please help me to figure out why \*bulk=True \*has no effect? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Alberto\_Paro\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alberto_paro_2/32/1137_2.png) [@Alberto\_Paro\_2](https://discuss.elastic.co/u/Alberto_Paro_2)\ **Post date:** [June 11, 2013, 5:13pm UTC](https://discuss.elastic.co/t/pyes-bulk-insert-problem/12356/2 "2013-06-11T17:13:05Z")
first you need to pass bulk\_size on connection creation refresh is not required. do it at the end of indexing process. before bulk indexing reduce the refresh\_interval to -1 in the index settings. restore it at the end. bulk\_index should work Inviato da iPhone Il giorno 11/giu/2013, alle ore 08:28, Dmitry Babitsky [dimok21@gmail.com](mailto:dimok21@gmail.com) ha scritto: > Hi, > > I have a 200M documents index I would like to reindex. > I wrote the following script that goes over documents in the old index and puts them with balk insert into the new index. > The size of each bulk is 2000 documents. > > search\_obj = pyes.query.Search(query = pyes.query.MatchAllQuery(), start=resume\_from) > > old\_index\_iterator = self.esconn.search(search\_obj, self.index\_name) > counter = 0 > BULK\_SIZE = 2000 > > for doc in old\_index\_iterator: > self.esconn.index(doc=doc, doc\_type=DOC\_TYPE, index=new\_index\_name, id=doc.get\_id(), bulk=True) > counter += 1 > > if counter % BULK\_SIZE == 0: > self.logger.debug("Refreshing...") > self.esconn.refresh() > self.logger.debug("Refresh done.") > > self.esconn.refresh() > > Observation: > > 1. The speed that I get is very slow: around 150 documents / minute. > 2. The time of the refresh operation is 0. > 3. If I remove the index command (just read from the DB) - I speed up 10 times. > > Conclusion: > The index ignores the bulk=True flag, and pushes every single document to the ES server. > > ## Anyone know please help me to figure out why bulk=True has no effect? > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Dmitry\_Babitsky](https://avatars.discourse-cdn.com/v4/letter/d/85e7bf/32.png) [@Dmitry\_Babitsky](https://discuss.elastic.co/u/Dmitry_Babitsky)\ **Post date:** [June 12, 2013, 9:52am UTC](https://discuss.elastic.co/t/pyes-bulk-insert-problem/12356/3 "2013-06-12T09:52:47Z")
Hi Alberto, I've implemented your suggestion (bulk\_size at connection creation + update\_settings on refresh\_interval to -1 on the new index), but unfortunately it did not help. On Tuesday, June 11, 2013 8:13:05 PM UTC+3, Alberto Paro wrote: > first you need to pass bulk\_size on connection creation > refresh is not required. do it at the end of indexing process. > before bulk indexing reduce the refresh\_interval to -1 in the index > settings. restore it at the end. > bulk\_index should work > > Inviato da iPhone > > Il giorno 11/giu/2013, alle ore 08:28, Dmitry Babitsky \<[dim...@gmail.com](mailto:dim...@gmail.com)\\> > ha scritto: > > Hi, > > I have a 200M documents index I would like to reindex. > I wrote the following script that goes over documents in the old index and > puts them with balk insert into the new index. > The size of each bulk is 2000 documents. > > search\_obj = pyes.query.Search(query = pyes.query.MatchAllQuery(), start= > resume\_from) > > old\_index\_iterator = self.esconn.search(search\_obj, self.index\_name) > counter = 0 > BULK\_SIZE = 2000 > > for doc in old\_index\_iterator: > self.esconn._index_(doc=doc, doc\_type=DOC\_TYPE, index=new\_index\_name,id > =doc.get\_id(),\* bulk=True\*) > counter += 1 > > if counter % BULK\_SIZE == 0: > self.logger.debug("Refreshing...") > > - > > ``` > **self.esconn.refresh()** > > ``` > > - > > ``` > self.logger.debug("Refresh done.") > > ``` > > self.esconn.refresh() > > Observation: > > 1. The speed that I get is very slow: around 150 documents / minute. > 2. The time of the refresh operation is 0. > 3. If I remove the index command (just read from the DB) - I speed up 10 > times. > > Conclusion: > The index ignores the \*bulk=True \*flag, and pushes every single document > to the ES server. > > Anyone know please help me to figure out why \*bulk=True \*has no effect? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:31am UTC](https://discuss.elastic.co/t/pyes-bulk-insert-problem/12356/4 "2017-07-06T02:31:41Z")