# Reindexing via scan type - revisited **URL:** \ **Category:** Elasticsearch\ **Created:** [February 12, 2013, 11:38pm UTC](https://discuss.elastic.co/t/reindexing-via-scan-type-revisited/10719 "2013-02-12T23:38:42Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![es\_learner](https://avatars.discourse-cdn.com/v4/letter/e/8dc957/32.png) [@es\_learner](https://discuss.elastic.co/u/es_learner)\ **Post date:** [February 12, 2013, 11:38pm UTC](https://discuss.elastic.co/t/reindexing-via-scan-type-revisited/10719/1 "2013-02-12T23:38:42Z")
I read this thread - [http://elasticsearch-users.115913.n3.nabble.com/reindexing-via-scan-search-type-td3256635.html#a3259255](http://elasticsearch-users.115913.n3.nabble.com/reindexing-via-scan-search-type-td3256635.html#a3259255) and have a few more questions. I'm using pyes(0.16.0) client running against ES server(0.19.x). The installation I am working on has 3 different indices, the biggest of which has about 50M docs. 3 nodes, each hosting the 3 indices and replicas. i.e. each index replicated to other 2 nodes. 5 shards. The installation is PROD, serving live read/write traffic. Because business SLA has changed and a number of fields have been deprecated, I would like to 'remove' these fields and 'compact' the indices without changing the underlying mapping. But instead of a new mapping and indexing from scratch, I was hoping to do a 'rolling' reindex of 'dehydrated' docs. Python code: batch\_size = 5 n = 0 while True: docs = scan\_index(start=n, size=batch\_size) # get next batch of docs from index if not docs: break n += len(docs) new\_docs = dehydrate(docs) # create new doc out of old doc sans deprecated fields reindex(new\_docs) # reindex new docs. This involves a delete(), followed by index() time.sleep(0.5) From the previous thread, I'm aware of potential issues with delete's but delete's are necessary before reindex can take place. In testing my 'dehydrate' Python code, I first created test indices of about 50 docs with the old mapping. A run with size=10 would simulate 5 batch reads/scan's and 5 batch reindex's. Problem(which I kind of anticipated): a) scan\_index() 'misses' some docs. However, multiple runs will eventually catch most if not all docs. I suspect my 'scan' logic is faulty and would appreciate comments. Please bear with me with (abridged copy) of my Python and pyes client codes: from pyes import ES def scan\_es(start, size): connection = ES(['localhost:9200']) query = {'query':{'match\_all':{}}, 'from':start, 'size':size} ``` resultset = connection.search(query, indexes=['test_index'], scan=True) for r in resultset['hits']['hits']: yield r['_source'] ``` def scan\_index(start, size): docs = [] for d in scan\_es(start, size): docs.append(d) return docs def reindex(docs): connection = ES(['localhost:9200']) for doc in docs: try: connection.get('test\_index', 'test\_doc', doc['id'], fields=['\_id']) connection.delete('test\_index', 'test\_doc', doc['id']) connection.index(doc, 'test\_index', 'test\_doc', doc['id']) except: pass Questions: 1. So each batch would invoke a new call to scan\_index/scan\_es. How does the scroll\_id get managed? I looked at github pyes code and the 'scan=True' seem to automagically manage the scroll\_id but I didn't dig deep enough to figure out if it gets reset on each search() call. 2. There is a pyes search\_scroll() API but I have no idea how to manage the scroll\_id. Advice pls. 3. The rolling reindex is expected to take a few days which is intentional so as not to slam PROD nodes. I can only increase batch\_size to maybe 10,000 and continue to be nice to the nodes. 4. What would be a more elegant way of achieving this? ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [February 13, 2013, 10:36am UTC](https://discuss.elastic.co/t/reindexing-via-scan-type-revisited/10719/2 "2013-02-13T10:36:57Z")
> Questions: > > 1. So each batch would invoke a new call to scan\_index/scan\_es. How does > the scroll\_id get managed? I looked at github pyes code and the 'scan=True' > seem to automagically manage the scroll\_id but I didn't dig deep enough to > figure out if it gets reset on each search() call. Looking at the pyes code, it looks like, if you set 'scan' then it automatically sets a scroll time as well. However, you shouldn't repeat the search. Instead, you should extract the scroll ID from each previous request, and pass it (plus a scroll time) to /\_search/scroll. Aparo actually provides the loop to do that here: > > 1. The rolling reindex is expected to take a few days which is intentional > so as not to slam PROD nodes. I can only increase batch\_size to maybe > 10,000 and continue to be nice to the nodes. 10,000 is a lot. Also consider that the number of results returned = $size \* $number\_of\_shards. So with 5 shards, and a size of 10,000 you'd get back 50,000 results. I find that the sweet spot is somewhere between 500 and 5,000 (depending on doc size, RAM etc) clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![es\_learner](https://avatars.discourse-cdn.com/v4/letter/e/8dc957/32.png) [@es\_learner](https://discuss.elastic.co/u/es_learner)\ **Post date:** [February 14, 2013, 9:28pm UTC](https://discuss.elastic.co/t/reindexing-via-scan-type-revisited/10719/3 "2013-02-14T21:28:05Z")
Thanks, Clinton. I'll give that a shot. A few more questions: a) Is the scroll id consistent (immutable)? I'm thinking of a scenario where I can have cron jobs running every midnight for say 5 hours, save the last scroll id and exit. Subsequent jobs pick up from the last scroll id. Rinse, repeat. All the while the live index continues getting new docs as well as reindexed docs. b) Do the 2nd and subsequent jobs calls search\_scroll, bypassing the search(...,scan=True)? If not, set the start=last\_scroll\_id on each restart followed by search\_scroll()? Thanks, dave ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [February 21, 2013, 1:18pm UTC](https://discuss.elastic.co/t/reindexing-via-scan-type-revisited/10719/4 "2013-02-21T13:18:22Z")
Hi Dave > A few more questions: > a) Is the scroll id consistent (immutable)? I'm thinking of a scenario > where I can have cron jobs running every midnight for say 5 hours, save the > last scroll id and exit. Subsequent jobs pick up from the last scroll id. > Rinse, repeat. All the while the live index continues getting new docs as > well as reindexed docs. The scroll ID can change on each scroll request, so for each subsequent scroll request, you need to use the scroll ID from the previous request. The scroll ID will expire 'scroll' time after the last request. Eg if you set 'scroll' to '1m', then one minute after the previous request, the scroll ID will expire. You need to set the 'scroll' param on each scroll request as well. You don't want to keep these around too long, so don't set the scroll param to eg '1d'. It stops old segments from being cleaned up. > b) Do the 2nd and subsequent jobs calls search\_scroll, bypassing the > search(...,scan=True)? If not, set the start=last\_scroll\_id on each restart > followed by search\_scroll()? The first request should be to /\_search and subsequent requests to /\_search/scroll clint > Thanks, > dave > > -- > View this message in context: [http://elasticsearch-users.115913.n3.nabble.com/Reindexing-via-scan-type-revisited-tp4029731p4029849.html](http://elasticsearch-users.115913.n3.nabble.com/Reindexing-via-scan-type-revisited-tp4029731p4029849.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![es\_learner](https://avatars.discourse-cdn.com/v4/letter/e/8dc957/32.png) [@es\_learner](https://discuss.elastic.co/u/es_learner)\ **Post date:** [March 12, 2013, 4:41am UTC](https://discuss.elastic.co/t/reindexing-via-scan-type-revisited/10719/5 "2013-03-12T04:41:27Z")
Thanks again, Clinton. I have been testing against a staging cluster of ~1M docs and it works well. scroll was set to '10m' and it took about 1 hr to scan and reindex 150,000 docs, across network traffic. Question: You mention "don't set the scroll param to eg '1d'. It stops old segments from being cleaned up." What is the mechanics behind this? For example, in my above testing, with a scroll=10m and the system running continuously for a few hours, will that affect cleanup's? What about read's and write's from other clients? Will write's get blocked and will read's see stale data for those few hours? I'm preparing for PROD run's against live data and traffic and the total number of docs is about 45M docs. What advice would you give to tackle this scan-and-reindex process? My plan was to run 1M at a time during low traffic hours between 10pm to 5am PST. This conservative approach will take me 45 days! Or I could run it 24x7 which would take ~5days. Edit: My concern is with the read's and write's locks if any. CPU-wise, with some throttling, the additional load is 15% and well within tolerance. ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [March 12, 2013, 10:23am UTC](https://discuss.elastic.co/t/reindexing-via-scan-type-revisited/10719/6 "2013-03-12T10:23:14Z")
Hiya > I have been testing against a staging cluster of ~1M docs and it works well. > scroll was set to '10m' and it took about 1 hr to scan and reindex 150,000 > docs, across network traffic. > > Question: You mention "don't set the scroll param to eg '1d'. It stops old > segments from being cleaned up." > What is the mechanics behind this? For example, in my above testing, with a > scroll=10m and the system running continuously for a few hours, will that > affect cleanup's? What about read's and write's from other clients? Will > write's get blocked and will read's see stale data for those few hours? I'm > preparing for PROD run's against live data and traffic and the total number > of docs is about 45M docs. What advice would you give to tackle this > scan-and-reindex process? My plan was to run 1M at a time during low > traffic hours between 10pm to 5am PST. This conservative approach will take > me 45 days! Or I could run it 24x7 which would take ~5days. As you index new documents, ES writes new "segments", where a segment is like a fully functional inverted index all by itself. When you do a search, ES searches through all the current segments, one by one. Every second, ES refreshes its view on search. That is, it opens readers against all the current segments. As more segments get written, so ES will merge eg 3 smaller segments into 1 new bigger segment. Normally these old segments are deleted, and ES starts searching in the new segment. When you specify that you want a scroll, ES takes a snapshot of the current segments, and remembers them. So the results from your scroll request always reflect the results as they were at that point in time. Results for the scroll request won't change as you keep indexing. However, new search requests WILL see the new segments and will return fresh data. When merges happen, the segments involved in a scroll request aren't deleted. They stick around until the scroll is finished, or the scroll timeout is reached. The timeout is renewed every time you pull another tranche of results from the scroll request. So the only thing to keep in mind is that you end up having many more segments open than usual, which can use up file descriptors, and memory etc. Make sure you have enough of both to last the whole time required to finish your reindex. clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:47am UTC](https://discuss.elastic.co/t/reindexing-via-scan-type-revisited/10719/7 "2017-07-06T02:47:01Z")