# Reindexing via scan search type **URL:** \ **Category:** Elasticsearch\ **Created:** [August 15, 2011, 7:08pm UTC](https://discuss.elastic.co/t/reindexing-via-scan-search-type/5164 "2011-08-15T19:08:19Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![Curtis\_Caravone](https://avatars.discourse-cdn.com/v4/letter/c/e47c2d/32.png) [@Curtis\_Caravone](https://discuss.elastic.co/u/Curtis_Caravone)\ **Post date:** [August 15, 2011, 7:08pm UTC](https://discuss.elastic.co/t/reindexing-via-scan-search-type/5164/1 "2011-08-15T19:08:19Z")
Hey, Sorry if this has already been answered... When executing a "scan" search type query, what happens to documents which have been indexed since the scroll started? Do they get added to the eventual result set or do you only scroll through a snapshot of the data as it existed at search time? I'm mainly interested in the feasibility of, say adding a new indexed field to a mapping and then reindexing the existing data using the scan search type. I realize I could alternatively do tricks with new indices and shifting aliases, but this seemed like a simpler approach. Curtis ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 15, 2011, 11:24pm UTC](https://discuss.elastic.co/t/reindexing-via-scan-search-type/5164/2 "2011-08-15T23:24:52Z")
You won't see new documents indexed after the first scan operation has "started", its a point in time view of the data. On Mon, Aug 15, 2011 at 10:08 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > Hey, > > Sorry if this has already been answered... When executing a "scan" search > type query, what happens to documents which have been indexed since the > scroll started? Do they get added to the eventual result set or do you only > scroll through a snapshot of the data as it existed at search time? > > I'm mainly interested in the feasibility of, say adding a new indexed field > to a mapping and then reindexing the existing data using the scan search > type. > > I realize I could alternatively do tricks with new indices and shifting > aliases, but this seemed like a simpler approach. > > Curtis ---
**Author:** ![James\_Cook](https://avatars.discourse-cdn.com/v4/letter/j/898d66/32.png) [@James\_Cook](https://discuss.elastic.co/u/James_Cook)\ **Post date:** [August 15, 2011, 11:35pm UTC](https://discuss.elastic.co/t/reindexing-via-scan-search-type/5164/3 "2011-08-15T23:35:14Z")
Is it truly a point in time view of the data, or are the ids just cached? If it was a PiT snapshot that would mean my backup problem that I thought I had, isn't a problem. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 16, 2011, 1:12am UTC](https://discuss.elastic.co/t/reindexing-via-scan-search-type/5164/4 "2011-08-16T01:12:55Z")
Its a point in time snapshot, taken on the first scroll request. No caching. On Tue, Aug 16, 2011 at 2:35 AM, James Cook [jcook@tracermedia.com](mailto:jcook@tracermedia.com) wrote: > Is it truly a point in time view of the data, or are the ids just cached? > If it was a PiT snapshot that would mean my backup problem that I thought I > had, isn't a problem. ---
**Author:** ![James\_Cook](https://avatars.discourse-cdn.com/v4/letter/j/898d66/32.png) [@James\_Cook](https://discuss.elastic.co/u/James_Cook)\ **Post date:** [August 16, 2011, 1:23am UTC](https://discuss.elastic.co/t/reindexing-via-scan-search-type/5164/5 "2011-08-16T01:23:15Z")
Hi Shay, Sorry for being dense, but I don't understand what "point in time" and "no caching" means here. Take this scenario: 1. I issue a scan query 2. I begin to scroll thru result pages 3. A document id re-indexed (and I haven't yet scolled to that document) 4. I eventually get to that newly updated document. Will I see the version of the document at the time I issued the scan query, or will I see the updated document? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 16, 2011, 1:29am UTC](https://discuss.elastic.co/t/reindexing-via-scan-search-type/5164/6 "2011-08-16T01:29:25Z")
No problem :). You will see the document version at the time you issued the scan request. On Tue, Aug 16, 2011 at 4:23 AM, James Cook [jcook@tracermedia.com](mailto:jcook@tracermedia.com) wrote: > Hi Shay, > > Sorry for being dense, but I don't understand what "point in time" and "no > caching" means here. > > Take this scenario: > > 1. I issue a scan query > 2. I begin to scroll thru result pages > 3. A document id re-indexed (and I haven't yet scolled to that > document) > 4. I eventually get to that newly updated document. > > Will I see the version of the document at the time I issued the scan query, > or will I see the updated document? ---
**Author:** ![James\_Cook](https://avatars.discourse-cdn.com/v4/letter/j/898d66/32.png) [@James\_Cook](https://discuss.elastic.co/u/James_Cook)\ **Post date:** [August 16, 2011, 1:43am UTC](https://discuss.elastic.co/t/reindexing-via-scan-search-type/5164/7 "2011-08-16T01:43:00Z")
That is incredible (I think). I was investigating snapshotting file systems so I could get a consistent copy of my datastore when I do backups. What you are basically saying is that somehow, ES is taking care of that for me. I could see this might be the case if you were caching the id's and version numbers at the time of the scan request, but that would also require retrieval of document by version number which I don't believe exists. My second guess would be that you write only to the transaction log during the scan's scroll lifetime, but that migt only allow for a single scan at any time. I didn't see details of what is going on under the hood in the docs[http://www.elasticsearch.org/guide/reference/api/search/search-type.html](http://www.elasticsearch.org/guide/reference/api/search/search-type.html), blog [http://www.elasticsearch.org/blog/2011/03/24/new-search-types.html](http://www.elasticsearch.org/blog/2011/03/24/new-search-types.html) or on the issue [https://github.com/elasticsearch/elasticsearch/issues/707](https://github.com/elasticsearch/elasticsearch/issues/707). Can you share how you the internals work? ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [August 16, 2011, 5:52am UTC](https://discuss.elastic.co/t/reindexing-via-scan-search-type/5164/8 "2011-08-16T05:52:48Z")
Hi James, I think Shay touched some of this aspects in his BBUZZ 2011 talk a bit, but as far as I understand this is relatively easily possible due to how Lucene works internally. When you update a document then it is in fact deleted (marked as deleted) in the index and a new document (new version of that document) is added to the index. [I do not consider myself an Lucene expert but the following is how I \*think\* it could work, but may be Shay uses different approach] This means that when the scroll starts ES probably gets an IndexReader and once IndexReader is opened it does not take account on index changes which occurred after it was opened (one has to reopen the IndexReader for it to see new changes). When the document is deleted then it is not removed from the index file immediately, it is just marked as deleted (which I think means that for example you can still get stored field values from deleted document should they be needed for a search results details that were generated before the document was deleted) and when a new documents are added then they are added to a new segment file (and this segment did not exist when the IndexReader was opened). [However, I am not sure how this fits with segment merges, because merges remove whole segments and generate a new one so this could possible be in conflict with a long scroll... dunno how exactly this is solved... may be merges are postponed while scrolls are opened?] Regards, Lukas On Tue, Aug 16, 2011 at 3:43 AM, James Cook [jcook@tracermedia.com](mailto:jcook@tracermedia.com) wrote: > That is incredible (I think). I was investigating snapshotting file systems > so I could get a consistent copy of my datastore when I do backups. What you > are basically saying is that somehow, ES is taking care of that for me. I > could see this might be the case if you were caching the id's and version > numbers at the time of the scan request, but that would also require > retrieval of document by version number which I don't believe exists. > > My second guess would be that you write only to the transaction log during > the scan's scroll lifetime, but that migt only allow for a single scan at > any time. > > I didn't see details of what is going on under the hood in the docs[http://www.elasticsearch.org/guide/reference/api/search/search-type.html](http://www.elasticsearch.org/guide/reference/api/search/search-type.html), > blog [http://www.elasticsearch.org/blog/2011/03/24/new-search-types.html](http://www.elasticsearch.org/blog/2011/03/24/new-search-types.html)or on the > issue [https://github.com/elasticsearch/elasticsearch/issues/707](https://github.com/elasticsearch/elasticsearch/issues/707). Can you > share how you the internals work? ---
**Author:** ![Paul\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_smith/32/1323_2.png) [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)\ **Post date:** [August 16, 2011, 7:04am UTC](https://discuss.elastic.co/t/reindexing-via-scan-search-type/5164/9 "2011-08-16T07:04:24Z")
dusting off my Lucene internals, I may not be 100% correct here, Shay/Otis etc will find a hole here probably, but In Lucene, every new item is a new segment, it's easy for point in time. one's reader just doesn't include the new segments. As new items are added in Lucene, they reach a merge threshold and then merge to a brand new, large segment. If your merge factor is 3, then as each item is added ,you get this sort of pattern (where each number is a segment holding X items inside it) 1 1 1 1 1 1 3 3 1 3 1 1 6 6 1 6 1 1 9 etc. The trick here is that a merge is still always a new file, the older segments are 'discarded' (marked as deleted in the Filesystem, but if you have an open process holding references to that file you can still read it through the file descriptor, it's just no longer in the filesystem directory table) The point in time is that your Reader is just holding references to the segments at that point in time. New segments created you don't see. even in the merge case (1 1 1 -\> 3) the old reader holds references to the 3x1, even after they're merged to a single '3'. Deletes are treated specially by not actually deleting, but tracking a 'delete list' in a separate internal lucene file. the point in time reader won't see any changes to that file (it's small enough to keep in memory), so you still maintain that state. Paul Smith ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 16, 2011, 10:51am UTC](https://discuss.elastic.co/t/reindexing-via-scan-search-type/5164/10 "2011-08-16T10:51:42Z")
Merge segment merging and flushing has advanced a bit :), but the mechanism to keep a point in time view is the same as explained. An IndexReader in lucene "sees" only the data that was there when it was opened (the segments it works against). On Tue, Aug 16, 2011 at 10:04 AM, Paul Smith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) wrote: > dusting off my Lucene internals, I may not be 100% correct here, Shay/Otis > etc will find a hole here probably, but In Lucene, every new item is a new > segment, it's easy for point in time. one's reader just doesn't include the > new segments. As new items are added in Lucene, they reach a merge > threshold and then merge to a brand new, large segment. > > If your merge factor is 3, then as each item is added ,you get this sort of > pattern (where each number is a segment holding X items inside it) > > 1 > 1 1 > 1 1 1 > 3 > 3 1 > 3 1 1 > 6 > 6 1 > 6 1 1 > 9 > > etc. > > The trick here is that a merge is still always a new file, the older > segments are 'discarded' (marked as deleted in the Filesystem, but if you > have an open process holding references to that file you can still read it > through the file descriptor, it's just no longer in the filesystem directory > table) > > The point in time is that your Reader is just holding references to the > segments at that point in time. New segments created you don't see. even > in the merge case (1 1 1 -\> 3) the old reader holds references to the 3x1, > even after they're merged to a single '3'. Deletes are treated specially by > not actually deleting, but tracking a 'delete list' in a separate internal > lucene file. the point in time reader won't see any changes to that file > (it's small enough to keep in memory), so you still maintain that state. > > Paul Smith ---
**Author:** ![Curtis\_Caravone](https://avatars.discourse-cdn.com/v4/letter/c/e47c2d/32.png) [@Curtis\_Caravone](https://discuss.elastic.co/u/Curtis_Caravone)\ **Post date:** [August 16, 2011, 4:38pm UTC](https://discuss.elastic.co/t/reindexing-via-scan-search-type/5164/11 "2011-08-16T16:38:45Z")
Thanks, that's perfect. So, would this be a safe reindexing pattern? - Put new mapping (all docs from this point on get new mapping) - Refresh (Make sure we see all the old docs) - Query all\_docs with scan search type - Reindex (update) each doc in the result (skip if I get a version conflict, since this means a new version already made it in) Once all this is done, all docs will have the new mapping applied? Curtis On Mon, Aug 15, 2011 at 4:24 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > You won't see new documents indexed after the first scan operation has > "started", its a point in time view of the data. > > On Mon, Aug 15, 2011 at 10:08 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > Hey, > > > > Sorry if this has already been answered... When executing a "scan" search > > type query, what happens to documents which have been indexed since the > > scroll started? Do they get added to the eventual result set or do you only > > scroll through a snapshot of the data as it existed at search time? > > > > I'm mainly interested in the feasibility of, say adding a new indexed > > field to a mapping and then reindexing the existing data using the scan > > search type. > > > > I realize I could alternatively do tricks with new indices and shifting > > aliases, but this seemed like a simpler approach. > > > > Curtis ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 16, 2011, 5:31pm UTC](https://discuss.elastic.co/t/reindexing-via-scan-search-type/5164/12 "2011-08-16T17:31:04Z")
Generally, yea, this scheme will work, but you might need to be careful with deletes happening... . On Tue, Aug 16, 2011 at 7:38 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com) wrote: > Thanks, that's perfect. So, would this be a safe reindexing pattern? > > - Put new mapping (all docs from this point on get new mapping) > - Refresh (Make sure we see all the old docs) > - Query all\_docs with scan search type > - Reindex (update) each doc in the result (skip if I get a version > conflict, since this means a new version already made it in) > > Once all this is done, all docs will have the new mapping applied? > > Curtis > > On Mon, Aug 15, 2011 at 4:24 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > You won't see new documents indexed after the first scan operation has > > "started", its a point in time view of the data. > > > > On Mon, Aug 15, 2011 at 10:08 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > Hey, > > > > > > Sorry if this has already been answered... When executing a "scan" > > > search type query, what happens to documents which have been indexed since > > > the scroll started? Do they get added to the eventual result set or do you > > > only scroll through a snapshot of the data as it existed at search time? > > > > > > I'm mainly interested in the feasibility of, say adding a new indexed > > > field to a mapping and then reindexing the existing data using the scan > > > search type. > > > > > > I realize I could alternatively do tricks with new indices and shifting > > > aliases, but this seemed like a simpler approach. > > > > > > Curtis ---
**Author:** ![Curtis\_Caravone](https://avatars.discourse-cdn.com/v4/letter/c/e47c2d/32.png) [@Curtis\_Caravone](https://discuss.elastic.co/u/Curtis_Caravone)\ **Post date:** [August 16, 2011, 6:37pm UTC](https://discuss.elastic.co/t/reindexing-via-scan-search-type/5164/13 "2011-08-16T18:37:01Z")
Cool, thanks. Looks like attempting an update with versioning will fail (correctly) if the doc has been deleted, so I think that covers it. Of course, I suppose there is nothing to prevent a doc from being deleted and then created again with a version that matches the one expected by the update... Curtis On Tue, Aug 16, 2011 at 10:31 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > Generally, yea, this scheme will work, but you might need to be careful > with deletes happening... . > > On Tue, Aug 16, 2011 at 7:38 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > Thanks, that's perfect. So, would this be a safe reindexing pattern? > > > > - Put new mapping (all docs from this point on get new mapping) > > - Refresh (Make sure we see all the old docs) > > - Query all\_docs with scan search type > > - Reindex (update) each doc in the result (skip if I get a version > > conflict, since this means a new version already made it in) > > > > Once all this is done, all docs will have the new mapping applied? > > > > Curtis > > > > On Mon, Aug 15, 2011 at 4:24 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > You won't see new documents indexed after the first scan operation has > > > "started", its a point in time view of the data. > > > > > > On Mon, Aug 15, 2011 at 10:08 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > > > Hey, > > > > > > > > Sorry if this has already been answered... When executing a "scan" > > > > search type query, what happens to documents which have been indexed since > > > > the scroll started? Do they get added to the eventual result set or do you > > > > only scroll through a snapshot of the data as it existed at search time? > > > > > > > > I'm mainly interested in the feasibility of, say adding a new indexed > > > > field to a mapping and then reindexing the existing data using the scan > > > > search type. > > > > > > > > I realize I could alternatively do tricks with new indices and shifting > > > > aliases, but this seemed like a simpler approach. > > > > > > > > Curtis ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 16, 2011, 11:48pm UTC](https://discuss.elastic.co/t/reindexing-via-scan-search-type/5164/14 "2011-08-16T23:48:44Z")
Yea, but, there is a timeout on how long deletes will be kept around for version checks (you can't keep them indefinitely) which defaults to 60 seconds. The setting is called index.gc\_deletes and defaults to 60s. One thing that we can do is allow to change this setting in realtime (add it to the list of settings that can be dynamically changed by update settings API), so you can increase that when doing the scan reindexing process, and then lower it back down. Opened an issue: [Update Settings: Allow to dynamically set index.gc\_deletes setting · Issue #1251 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1251) On Tue, Aug 16, 2011 at 9:37 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com) wrote: > Cool, thanks. Looks like attempting an update with versioning will fail > (correctly) if the doc has been deleted, so I think that covers it. Of > course, I suppose there is nothing to prevent a doc from being deleted and > then created again with a version that matches the one expected by the > update... > > Curtis > > On Tue, Aug 16, 2011 at 10:31 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > Generally, yea, this scheme will work, but you might need to be careful > > with deletes happening... . > > > > On Tue, Aug 16, 2011 at 7:38 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > Thanks, that's perfect. So, would this be a safe reindexing pattern? > > > > > > - Put new mapping (all docs from this point on get new mapping) > > > - Refresh (Make sure we see all the old docs) > > > - Query all\_docs with scan search type > > > - Reindex (update) each doc in the result (skip if I get a version > > > conflict, since this means a new version already made it in) > > > > > > Once all this is done, all docs will have the new mapping applied? > > > > > > Curtis > > > > > > On Mon, Aug 15, 2011 at 4:24 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > You won't see new documents indexed after the first scan operation has > > > > "started", its a point in time view of the data. > > > > > > > > On Mon, Aug 15, 2011 at 10:08 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > > > > > Hey, > > > > > > > > > > Sorry if this has already been answered... When executing a "scan" > > > > > search type query, what happens to documents which have been indexed since > > > > > the scroll started? Do they get added to the eventual result set or do you > > > > > only scroll through a snapshot of the data as it existed at search time? > > > > > > > > > > I'm mainly interested in the feasibility of, say adding a new indexed > > > > > field to a mapping and then reindexing the existing data using the scan > > > > > search type. > > > > > > > > > > I realize I could alternatively do tricks with new indices and shifting > > > > > aliases, but this seemed like a simpler approach. > > > > > > > > > > Curtis ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:56am UTC](https://discuss.elastic.co/t/reindexing-via-scan-search-type/5164/15 "2017-07-06T03:56:48Z")