# How to reindex last bulk of docs of elasticsearch.helpers.parallel\_bulk?

**URL:** <https://discuss.elastic.co/t/how-to-reindex-last-bulk-of-docs-of-elasticsearch-helpers-parallel-bulk/66887>\
**Category:** Elasticsearch\
**Created:** [November 22, 2016, 3:54pm UTC](https://discuss.elastic.co/t/how-to-reindex-last-bulk-of-docs-of-elasticsearch-helpers-parallel-bulk/66887 "2016-11-22T15:54:56Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Moshe\_Sucaz](https://avatars.discourse-cdn.com/v4/letter/m/0ea827/32.png) [@Moshe\_Sucaz](https://discuss.elastic.co/u/Moshe_Sucaz)\
**Post date:** [November 22, 2016, 3:54pm UTC](https://discuss.elastic.co/t/how-to-reindex-last-bulk-of-docs-of-elasticsearch-helpers-parallel-bulk/66887/1 "2016-11-22T15:54:56Z")

</div>

Hi,  
I am upgrading our elastic from 1.5.2 to 2.3.4. I wrote python re index script that is using elasticsearch.helpers.parallel\_bulk.  
When I am running the script, its reindexing perfectly 99% of the docs, but the last bulk of docs is never reindexed.  
for example, if I have 2276383 docs to reindex, and I am running parallel\_bulk with chunk\_size=500, then its reindexing 2276000,  
and the last 383 files are not indexed and script never ends. How can I make it reindex last docs?

The function that I wrote to use parallel\_bulk is:

> def parallel\_reindex(index\_name, doc\_type, chunk\_size=500, scroll='10m', scan\_kwargs={}, bulk\_kwargs={}):  
> target\_client = Elasticsearch(hosts = ['node01:9200', 'node02:9200', 'node03:9200'], retry\_on\_timeout = True, max\_retries = 10, timeout = 1000)  
> source\_client = Elasticsearch(hosts = ['node04:9200', 'node05:9200', 'node06:9200'], retry\_on\_timeout=True, max\_retries=10, timeout=1000)  
> query = {"query": {"match\_all": {}}}  
> docs = scan(source\_client,  
> query = query,  
> index = index\_name,  
> scroll = scroll,  
> doc\_type=doc\_type,  
> \*\* scan\_kwargs  
> )

> def \_change\_doc\_params\_to\_elastic\_2(hits, target\_client):  
> for h in hits:  
> # make some changes  
> yield h  
> kwargs = {  
> 'stats\_only': True,  
> }  
> kwargs.update(bulk\_kwargs)  
> for response in parallel\_bulk(target\_client, \_change\_doc\_params\_to\_elastic\_2(docs, target\_client), thread\_count=8, chunk\_size=chunk\_size, max\_chunk\_bytes=20 \* 1014 \* 1024):  
> pass

> print("Done parallel\_reindex of doc\_type %s in index %s" % (doc\_type,index\_name))

Thanks

---

<div class="post-metadata">

**Author:** ![mainec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mainec/32/5557_2.png) [@mainec](https://discuss.elastic.co/u/mainec)\
**Post date:** [November 23, 2016, 10:52am UTC](https://discuss.elastic.co/t/how-to-reindex-last-bulk-of-docs-of-elasticsearch-helpers-parallel-bulk/66887/2 "2016-11-23T10:52:27Z")

</div>

I'm not particularly familiar with the Python client, so can't actually help you with your script.

What I am wondering though is whether the Reindex API that's built into 2.3.x would help you:

[https://www.elastic.co/guide/en/elasticsearch/reference/2.3/docs-reindex.html](https://www.elastic.co/guide/en/elasticsearch/reference/2.3/docs-reindex.html)

---

<div class="post-metadata">

**Author:** ![Moshe\_Sucaz](https://avatars.discourse-cdn.com/v4/letter/m/0ea827/32.png) [@Moshe\_Sucaz](https://discuss.elastic.co/u/Moshe_Sucaz)\
**Post date:** [November 30, 2016, 6:51am UTC](https://discuss.elastic.co/t/how-to-reindex-last-bulk-of-docs-of-elasticsearch-helpers-parallel-bulk/66887/3 "2016-11-30T06:51:56Z")

</div>

Thanks Isabel,  
When I started to work on the migration, I remember reading somewhere that the reindex API will not work for migration of 1.x to 2.x. I also have too many changes and I am indexing from 1.5.2 cluster to 2.3.4 new cluster, so I created my own script. Anyway, issue solved! It was something related to elasticsearcgh-py 2.x that doesn't support Elasticsearch version 1.x. (see: [https://github.com/elastic/elasticsearch-py/issues/438](https://github.com/elastic/elasticsearch-py/issues/438)). When I connected to 1.x cluster with [pyes.es.ES](http://pyes.es.ES), instead of elasticsearcgh-py, issue solved.

Thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 28, 2016, 6:52am UTC](https://discuss.elastic.co/t/how-to-reindex-last-bulk-of-docs-of-elasticsearch-helpers-parallel-bulk/66887/4 "2016-12-28T06:52:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.