Sorry if this has already been answered... When executing a "scan" search
type query, what happens to documents which have been indexed since the
scroll started? Do they get added to the eventual result set or do you only
scroll through a snapshot of the data as it existed at search time?
I'm mainly interested in the feasibility of, say adding a new indexed field
to a mapping and then reindexing the existing data using the scan search
type.
I realize I could alternatively do tricks with new indices and shifting
aliases, but this seemed like a simpler approach.
You won't see new documents indexed after the first scan operation has
"started", its a point in time view of the data.
On Mon, Aug 15, 2011 at 10:08 PM, Curtis Caravone caravone@gmail.comwrote:
Hey,
Sorry if this has already been answered... When executing a "scan" search
type query, what happens to documents which have been indexed since the
scroll started? Do they get added to the eventual result set or do you only
scroll through a snapshot of the data as it existed at search time?
I'm mainly interested in the feasibility of, say adding a new indexed field
to a mapping and then reindexing the existing data using the scan search
type.
I realize I could alternatively do tricks with new indices and shifting
aliases, but this seemed like a simpler approach.
Is it truly a point in time view of the data, or are the ids just cached? If
it was a PiT snapshot that would mean my backup problem that I thought I
had, isn't a problem.
Is it truly a point in time view of the data, or are the ids just cached?
If it was a PiT snapshot that would mean my backup problem that I thought I
had, isn't a problem.
That is incredible (I think). I was investigating snapshotting file systems
so I could get a consistent copy of my datastore when I do backups. What you
are basically saying is that somehow, ES is taking care of that for me. I
could see this might be the case if you were caching the id's and version
numbers at the time of the scan request, but that would also require
retrieval of document by version number which I don't believe exists.
My second guess would be that you write only to the transaction log during
the scan's scroll lifetime, but that migt only allow for a single scan at
any time.
I think Shay touched some of this aspects in his BBUZZ 2011 talk a bit, but
as far as I understand this is relatively easily possible due to how Lucene
works internally. When you update a document then it is in fact deleted
(marked as deleted) in the index and a new document (new version of that
document) is added to the index.
[I do not consider myself an Lucene expert but the following is how I
*think* it could work, but may be Shay uses different approach]
This means that when the scroll starts ES probably gets an IndexReader and
once IndexReader is opened it does not take account on index changes which
occurred after it was opened (one has to reopen the IndexReader for it to
see new changes). When the document is deleted then it is not removed from
the index file immediately, it is just marked as deleted (which I think
means that for example you can still get stored field values from deleted
document should they be needed for a search results details that were
generated before the document was deleted) and when a new documents are
added then they are added to a new segment file (and this segment did not
exist when the IndexReader was opened). [However, I am not sure how this
fits with segment merges, because merges remove whole segments and generate
a new one so this could possible be in conflict with a long scroll... dunno
how exactly this is solved... may be merges are postponed while scrolls are
opened?]
That is incredible (I think). I was investigating snapshotting file systems
so I could get a consistent copy of my datastore when I do backups. What you
are basically saying is that somehow, ES is taking care of that for me. I
could see this might be the case if you were caching the id's and version
numbers at the time of the scan request, but that would also require
retrieval of document by version number which I don't believe exists.
My second guess would be that you write only to the transaction log during
the scan's scroll lifetime, but that migt only allow for a single scan at
any time.
dusting off my Lucene internals, I may not be 100% correct here, Shay/Otis
etc will find a hole here probably, but In Lucene, every new item is a new
segment, it's easy for point in time. one's reader just doesn't include the
new segments. As new items are added in Lucene, they reach a merge
threshold and then merge to a brand new, large segment.
If your merge factor is 3, then as each item is added ,you get this sort of
pattern (where each number is a segment holding X items inside it)
1
1 1
1 1 1
3
3 1
3 1 1
6
6 1
6 1 1
9
etc.
The trick here is that a merge is still always a new file, the older
segments are 'discarded' (marked as deleted in the Filesystem, but if you
have an open process holding references to that file you can still read it
through the file descriptor, it's just no longer in the filesystem directory
table)
The point in time is that your Reader is just holding references to the
segments at that point in time. New segments created you don't see. even
in the merge case (1 1 1 -> 3) the old reader holds references to the 3x1,
even after they're merged to a single '3'. Deletes are treated specially by
not actually deleting, but tracking a 'delete list' in a separate internal
lucene file. the point in time reader won't see any changes to that file
(it's small enough to keep in memory), so you still maintain that state.
Merge segment merging and flushing has advanced a bit :), but the mechanism
to keep a point in time view is the same as explained. An IndexReader in
lucene "sees" only the data that was there when it was opened (the segments
it works against).
dusting off my Lucene internals, I may not be 100% correct here, Shay/Otis
etc will find a hole here probably, but In Lucene, every new item is a new
segment, it's easy for point in time. one's reader just doesn't include the
new segments. As new items are added in Lucene, they reach a merge
threshold and then merge to a brand new, large segment.
If your merge factor is 3, then as each item is added ,you get this sort of
pattern (where each number is a segment holding X items inside it)
1
1 1
1 1 1
3
3 1
3 1 1
6
6 1
6 1 1
9
etc.
The trick here is that a merge is still always a new file, the older
segments are 'discarded' (marked as deleted in the Filesystem, but if you
have an open process holding references to that file you can still read it
through the file descriptor, it's just no longer in the filesystem directory
table)
The point in time is that your Reader is just holding references to the
segments at that point in time. New segments created you don't see. even
in the merge case (1 1 1 -> 3) the old reader holds references to the 3x1,
even after they're merged to a single '3'. Deletes are treated specially by
not actually deleting, but tracking a 'delete list' in a separate internal
lucene file. the point in time reader won't see any changes to that file
(it's small enough to keep in memory), so you still maintain that state.
Thanks, that's perfect. So, would this be a safe reindexing pattern?
Put new mapping (all docs from this point on get new mapping)
Refresh (Make sure we see all the old docs)
Query all_docs with scan search type
Reindex (update) each doc in the result (skip if I get a version
conflict, since this means a new version already made it in)
Once all this is done, all docs will have the new mapping applied?
Curtis
On Mon, Aug 15, 2011 at 4:24 PM, Shay Banon kimchy@gmail.com wrote:
You won't see new documents indexed after the first scan operation has
"started", its a point in time view of the data.
On Mon, Aug 15, 2011 at 10:08 PM, Curtis Caravone caravone@gmail.comwrote:
Hey,
Sorry if this has already been answered... When executing a "scan" search
type query, what happens to documents which have been indexed since the
scroll started? Do they get added to the eventual result set or do you only
scroll through a snapshot of the data as it existed at search time?
I'm mainly interested in the feasibility of, say adding a new indexed
field to a mapping and then reindexing the existing data using the scan
search type.
I realize I could alternatively do tricks with new indices and shifting
aliases, but this seemed like a simpler approach.
Generally, yea, this scheme will work, but you might need to be careful with
deletes happening... .
On Tue, Aug 16, 2011 at 7:38 PM, Curtis Caravone caravone@gmail.com wrote:
Thanks, that's perfect. So, would this be a safe reindexing pattern?
Put new mapping (all docs from this point on get new mapping)
Refresh (Make sure we see all the old docs)
Query all_docs with scan search type
Reindex (update) each doc in the result (skip if I get a version
conflict, since this means a new version already made it in)
Once all this is done, all docs will have the new mapping applied?
Curtis
On Mon, Aug 15, 2011 at 4:24 PM, Shay Banon kimchy@gmail.com wrote:
You won't see new documents indexed after the first scan operation has
"started", its a point in time view of the data.
On Mon, Aug 15, 2011 at 10:08 PM, Curtis Caravone caravone@gmail.comwrote:
Hey,
Sorry if this has already been answered... When executing a "scan"
search type query, what happens to documents which have been indexed since
the scroll started? Do they get added to the eventual result set or do you
only scroll through a snapshot of the data as it existed at search time?
I'm mainly interested in the feasibility of, say adding a new indexed
field to a mapping and then reindexing the existing data using the scan
search type.
I realize I could alternatively do tricks with new indices and shifting
aliases, but this seemed like a simpler approach.
Cool, thanks. Looks like attempting an update with versioning will fail
(correctly) if the doc has been deleted, so I think that covers it. Of
course, I suppose there is nothing to prevent a doc from being deleted and
then created again with a version that matches the one expected by the
update...
Curtis
On Tue, Aug 16, 2011 at 10:31 AM, Shay Banon kimchy@gmail.com wrote:
Generally, yea, this scheme will work, but you might need to be careful
with deletes happening... .
On Tue, Aug 16, 2011 at 7:38 PM, Curtis Caravone caravone@gmail.comwrote:
Thanks, that's perfect. So, would this be a safe reindexing pattern?
Put new mapping (all docs from this point on get new mapping)
Refresh (Make sure we see all the old docs)
Query all_docs with scan search type
Reindex (update) each doc in the result (skip if I get a version
conflict, since this means a new version already made it in)
Once all this is done, all docs will have the new mapping applied?
Curtis
On Mon, Aug 15, 2011 at 4:24 PM, Shay Banon kimchy@gmail.com wrote:
You won't see new documents indexed after the first scan operation has
"started", its a point in time view of the data.
On Mon, Aug 15, 2011 at 10:08 PM, Curtis Caravone caravone@gmail.comwrote:
Hey,
Sorry if this has already been answered... When executing a "scan"
search type query, what happens to documents which have been indexed since
the scroll started? Do they get added to the eventual result set or do you
only scroll through a snapshot of the data as it existed at search time?
I'm mainly interested in the feasibility of, say adding a new indexed
field to a mapping and then reindexing the existing data using the scan
search type.
I realize I could alternatively do tricks with new indices and shifting
aliases, but this seemed like a simpler approach.
Yea, but, there is a timeout on how long deletes will be kept around for
version checks (you can't keep them indefinitely) which defaults to 60
seconds. The setting is called index.gc_deletes and defaults to 60s.
One thing that we can do is allow to change this setting in realtime (add it
to the list of settings that can be dynamically changed by update settings
API), so you can increase that when doing the scan reindexing process, and
then lower it back down.
On Tue, Aug 16, 2011 at 9:37 PM, Curtis Caravone caravone@gmail.com wrote:
Cool, thanks. Looks like attempting an update with versioning will fail
(correctly) if the doc has been deleted, so I think that covers it. Of
course, I suppose there is nothing to prevent a doc from being deleted and
then created again with a version that matches the one expected by the
update...
Curtis
On Tue, Aug 16, 2011 at 10:31 AM, Shay Banon kimchy@gmail.com wrote:
Generally, yea, this scheme will work, but you might need to be careful
with deletes happening... .
On Tue, Aug 16, 2011 at 7:38 PM, Curtis Caravone caravone@gmail.comwrote:
Thanks, that's perfect. So, would this be a safe reindexing pattern?
Put new mapping (all docs from this point on get new mapping)
Refresh (Make sure we see all the old docs)
Query all_docs with scan search type
Reindex (update) each doc in the result (skip if I get a version
conflict, since this means a new version already made it in)
Once all this is done, all docs will have the new mapping applied?
Curtis
On Mon, Aug 15, 2011 at 4:24 PM, Shay Banon kimchy@gmail.com wrote:
You won't see new documents indexed after the first scan operation has
"started", its a point in time view of the data.
On Mon, Aug 15, 2011 at 10:08 PM, Curtis Caravone caravone@gmail.comwrote:
Hey,
Sorry if this has already been answered... When executing a "scan"
search type query, what happens to documents which have been indexed since
the scroll started? Do they get added to the eventual result set or do you
only scroll through a snapshot of the data as it existed at search time?
I'm mainly interested in the feasibility of, say adding a new indexed
field to a mapping and then reindexing the existing data using the scan
search type.
I realize I could alternatively do tricks with new indices and shifting
aliases, but this seemed like a simpler approach.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.