# Error during Re-Indexing, chunk size?

**URL:** <https://discuss.elastic.co/t/error-during-re-indexing-chunk-size/50559>\
**Category:** Elasticsearch\
**Created:** [May 20, 2016, 1:15pm UTC](https://discuss.elastic.co/t/error-during-re-indexing-chunk-size/50559 "2016-05-20T13:15:11Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![zetian](https://avatars.discourse-cdn.com/v4/letter/z/f14d63/32.png) [@zetian](https://discuss.elastic.co/u/zetian)\
**Post date:** [May 20, 2016, 1:15pm UTC](https://discuss.elastic.co/t/error-during-re-indexing-chunk-size/50559/1 "2016-05-20T13:15:11Z")

</div>

I am using v2.1 and using the python re-indexing helper.  
The source index has 500491 documents and a size of 150.4gb  
average document about 300kb.

So I chunked the bulk insert to 50 document, set 'refresh\_interval' to -1 and let it go.

After 490212 documents I received this error:

Traceback (most recent call last):  
File "reindex\_cps\_docs.py", line 33, in   
chunk\_size=50  
File "c:\Python27\lib\site-packages\elasticsearch\helpers\__init_\_.py", line 353, in reindex  
chunk\_size=chunk\_size, \*\*kwargs)  
File "c:\Python27\lib\site-packages\elasticsearch\helpers\__init_\_.py", line 188, in bulk  
for ok, item in streaming\_bulk(client, actions, \*\*kwargs):  
File "c:\Python27\lib\site-packages\elasticsearch\helpers\__init_\_.py", line 159, in streaming\_bulk  
for bulk\_actions in _chunk\_actions(actions, chunk\_size, max\_chunk\_bytes, client.transport.serializer):  
File "c:\Python27\lib\site-packages\elasticsearch\helpers\_init_.py", line 53, in _chunk\_actions  
for action, data in actions:  
File "c:\Python27\lib\site-packages\elasticsearch\helpers\_init_.py", line 342, in _change\_doc\_index  
for h in hits:  
File "c:\Python27\lib\site-packages\elasticsearch\helpers\_init_.py", line 283, in scan  
resp = client.scroll(scroll\_id, scroll=scroll)  
File "c:\Python27\lib\site-packages\elasticsearch\client\[utils.py](http://utils.py)", line 69, in _wrapped  
return func(\*args, params=params, \*\*kwargs)  
File "c:\Python27\lib\site-packages\elasticsearch\client\_init_.py", line 662, in scroll  
params=params, body=body)  
File "c:\Python27\lib\site-packages\elasticsearch\[transport.py](http://transport.py)", line 307, in perform\_request  
status, headers, data = connection.perform\_request(method, url, params, body, ignore=ignore, timeout=timeout)  
File "c:\Python27\lib\site-packages\elasticsearch\connection\http\_urllib3.py", line 93, in perform\_request  
self.\_raise\_error(response.status, raw\_data)  
File "c:\Python27\lib\site-packages\elasticsearch\connection\[base.py](http://base.py)", line 105, in \_raise\_error  
raise HTTP\_EXCEPTIONS.get(status\_code, TransportError)(status\_code, error\_message, additional\_info)  
elasticsearch.exceptions.TransportError: TransportError(503, u'{"\_scroll\_id":"c2NhbjswOzE7dG90YWxfaGl0czoyOTMyMjk7","too  
k":1,"timed\_out":false,"\_shards":{"total":5,"successful":0,"failed":0},"hits":{"total":293229,"max\_score":0.0,"hits":[]}  
}')

---

<div class="post-metadata">

**Author:** ![zetian](https://avatars.discourse-cdn.com/v4/letter/z/f14d63/32.png) [@zetian](https://discuss.elastic.co/u/zetian)\
**Post date:** [May 20, 2016, 1:17pm UTC](https://discuss.elastic.co/t/error-during-re-indexing-chunk-size/50559/2 "2016-05-20T13:17:35Z")

</div>

I set:  
timeout=30,  
max\_retries=10,  
retry\_on\_timeout=True

I can lower the chunk size. What else can I try?

---

<div class="post-metadata">

**Author:** ![zetian](https://avatars.discourse-cdn.com/v4/letter/z/f14d63/32.png) [@zetian](https://discuss.elastic.co/u/zetian)\
**Post date:** [May 20, 2016, 4:08pm UTC](https://discuss.elastic.co/t/error-during-re-indexing-chunk-size/50559/3 "2016-05-20T16:08:14Z")

</div>

I raised the Scan timeout to 10m and after about 3 hours the  
retry gives the following error:

Traceback (most recent call last):  
File "reindex\_cps\_docs.py", line 34, in   
scroll='10m'  
File "c:\Python27\lib\site-packages\elasticsearch\helpers\__init_\_.py", line 353, in reindex  
chunk\_size=chunk\_size, \*\*kwargs)  
File "c:\Python27\lib\site-packages\elasticsearch\helpers\__init_\_.py", line 188, in bulk  
for ok, item in streaming\_bulk(client, actions, \*\*kwargs):  
File "c:\Python27\lib\site-packages\elasticsearch\helpers\__init_\_.py", line 160, in streaming\_bulk  
for result in _process\_bulk\_chunk(client, bulk\_actions, raise\_on\_exception, raise\_on\_error, \*\*kwargs):  
File "c:\Python27\lib\site-packages\elasticsearch\helpers\_init_.py", line 89, in \_process\_bulk\_chunk  
raise e  
elasticsearch.exceptions.TransportError: TransportError(503, u'cluster\_block\_exception')

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:50pm UTC](https://discuss.elastic.co/t/error-during-re-indexing-chunk-size/50559/4 "2017-07-05T22:50:05Z")

</div>