The versioning feature allows to handle conflicts when doing a get/search
and then index/delete. Each document has a version number, and each index
and delete (tombstone) increase the version number.
A conflict is detected when providing a version parameter to the
index/delete request and there is a mismatch between the version provided
and the real time version of the current document. If a version number is
not provided, then the operation is forced without doing any version check.
Search and get operations return the version number associated with the hit
under _version. Note, the version number reflects the near real time
status of the search. In other words, the version number will point to the
doc when the index was last refreshed.
Nice side affects of this feature include returning on delete if the
document was actually found or not, as well as fixing a rare out of order
replication sync between a primary and its replica (under heavy changes of
the same doc within a short period of time).
The bulk operation supports providing a version number as well using _version on each bulk item.
Upgrade Notes
The versioning support is backward compatible, though only new operations on
documents will cause the versioning system to start and kick in for that
doc.
great feature,nice new year gift ,thanks for the hard working,i can see the time machine is coming~~
so cool!!
From: Shay Banon
Sent: Tuesday, January 04, 2011 10:07 AM
To: users
Subject: Just Pushed: Versioning
The versioning feature allows to handle conflicts when doing a get/search and then index/delete. Each document has a version number, and each index and delete (tombstone) increase the version number.
A conflict is detected when providing a version parameter to the index/delete request and there is a mismatch between the version provided and the real time version of the current document. If a version number is not provided, then the operation is forced without doing any version check.
Search and get operations return the version number associated with the hit under _version. Note, the version number reflects the near real time status of the search. In other words, the version number will point to the doc when the index was last refreshed.
Nice side affects of this feature include returning on delete if the document was actually found or not, as well as fixing a rare out of order replication sync between a primary and its replica (under heavy changes of the same doc within a short period of time).
The bulk operation supports providing a version number as well using _version on each bulk item.
Upgrade Notes
The versioning support is backward compatible, though only new operations on documents will cause the versioning system to start and kick in for that doc.
This is great Shay, an optimistic locking pattern usage. I like it.
This reminded me of an old topic I had raised a while back: http://elasticsearch-users.115913.n3.nabble.com/Index-Verification-td1430219.html.
We're beginning to use ES for some production support tools at the
moment,
but the real goal is to surgically remove the cancer that is our own home
grown index infrastructure (going to take a while).
Relating to Versioning stuff, but is that field an automatically stored
field? In an ideal case I'd love an API to hit to return all _id and
_version values for all items in a given index, I can drop the sort need
from the original post, as I can probably do that outside ES. I'm after the fastest stream of these 2 fields from an index to be compared with another
stream coming from the originating data source (our db) to validate the info
in ES.
I notice that this Versioning should address Issue # 490? (491 I agree
should have the _version information provided by the client, allow that to
worry about what clock to use, in our case, our DB is the 'primary source of
truth' and it generates a trigger-based lastupdated timestamp for the row).
The versioning feature allows to handle conflicts when doing a get/search
and then index/delete. Each document has a version number, and each index
and delete (tombstone) increase the version number.
A conflict is detected when providing a version parameter to the
index/delete request and there is a mismatch between the version provided
and the real time version of the current document. If a version number is
not provided, then the operation is forced without doing any version check.
Search and get operations return the version number associated with the hit
under _version. Note, the version number reflects the near real time
status of the search. In other words, the version number will point to the
doc when the index was last refreshed.
Nice side affects of this feature include returning on delete if the
document was actually found or not, as well as fixing a rare out of order
replication sync between a primary and its replica (under heavy changes of
the same doc within a short period of time).
The bulk operation supports providing a version number as well using _version on each bulk item.
Upgrade Notes
The versioning support is backward compatible, though only new operations
on documents will cause the versioning system to start and kick in for that
doc.
The versioning feature allows to handle conflicts when doing a get/search
and then index/delete. Each document has a version number, and each index
and delete (tombstone) increase the version number.
Hi,
This is a great feature. Even CouchDB has a '_rev' field which it uses
to maintain versions between documents.
Do you think it would be a good idea to map the CouchDB '_rev' field
to the ES '_version' field ? Specially in the 'CouchDB river' plugin.
If so, I can do that.
The versioning is not stored as a field, but you get it back when you do a
"get" or when searching (per hit). Regarding getting back all the hits for a
query (can be match_all), you can use search, with scrolling. The problem is
that with large result sets, it can get very expensive, even with scrolling.
I am thinking of adding a search_type called "scan" that will support
something like that, without the requirement to do any type of sorting
(score or other fields).
This is great Shay, an optimistic locking pattern usage. I like it.
This reminded me of an old topic I had raised a while back: http://elasticsearch-users.115913.n3.nabble.com/Index-Verification-td1430219.html. We're beginning to use ES for some production support tools at the moment,
but the real goal is to surgically remove the cancer that is our own home
grown index infrastructure (going to take a while).
Relating to Versioning stuff, but is that field an automatically stored
field? In an ideal case I'd love an API to hit to return all _id and
_version values for all items in a given index, I can drop the sort need
from the original post, as I can probably do that outside ES. I'm after the fastest stream of these 2 fields from an index to be compared with another
stream coming from the originating data source (our db) to validate the info
in ES.
I notice that this Versioning should address Issue # 490? (491 I agree
should have the _version information provided by the client, allow that to
worry about what clock to use, in our case, our DB is the 'primary source of
truth' and it generates a trigger-based lastupdated timestamp for the row).
The versioning feature allows to handle conflicts when doing a get/search
and then index/delete. Each document has a version number, and each index
and delete (tombstone) increase the version number.
A conflict is detected when providing a version parameter to the
index/delete request and there is a mismatch between the version provided
and the real time version of the current document. If a version number is
not provided, then the operation is forced without doing any version check.
Search and get operations return the version number associated with the
hit under _version. Note, the version number reflects the near real time
status of the search. In other words, the version number will point to the
doc when the index was last refreshed.
Nice side affects of this feature include returning on delete if the
document was actually found or not, as well as fixing a rare out of order
replication sync between a primary and its replica (under heavy changes of
the same doc within a short period of time).
The bulk operation supports providing a version number as well using _version on each bulk item.
Upgrade Notes
The versioning support is backward compatible, though only new operations
on documents will cause the versioning system to start and kick in for that
doc.
You can't use that since the version is internal to ES, and you can't index
a doc and control its version (if you provide the version, it will fail
since there is no such version, or it does not exists, the latter, to handle
deletes).
I don't think that you need to have that mapping with the couchdb
integration anyhow. Couchdb will maintain the correct stream of changes and
it will be applied to ES. Maybe you are after indexing the _rev as an
addition field in the json, so you can get it back when you search? If thats
the case, I think we can enhance the couchdb river to support that.
The versioning feature allows to handle conflicts when doing a get/search
and then index/delete. Each document has a version number, and each index
and delete (tombstone) increase the version number.
Hi,
This is a great feature. Even CouchDB has a '_rev' field which it uses
to maintain versions between documents.
Do you think it would be a good idea to map the CouchDB '_rev' field
to the ES '_version' field ? Specially in the 'CouchDB river' plugin.
If so, I can do that.
Two more benefits of versioning that I forgot to mention:
Index with create flag will now fail if there is already an indexed doc.
This basically allows to use create as a putIfAbsent logic.
Indexing a document that does not exists will be as fast as with create
flag set to true. So no need to set it anymore to improve fresh data
indexing performance.
The versioning feature allows to handle conflicts when doing a get/search
and then index/delete. Each document has a version number, and each index
and delete (tombstone) increase the version number.
A conflict is detected when providing a version parameter to the
index/delete request and there is a mismatch between the version provided
and the real time version of the current document. If a version number is
not provided, then the operation is forced without doing any version check.
Search and get operations return the version number associated with the hit
under _version. Note, the version number reflects the near real time
status of the search. In other words, the version number will point to the
doc when the index was last refreshed.
Nice side affects of this feature include returning on delete if the
document was actually found or not, as well as fixing a rare out of order
replication sync between a primary and its replica (under heavy changes of
the same doc within a short period of time).
The bulk operation supports providing a version number as well using _version on each bulk item.
Upgrade Notes
The versioning support is backward compatible, though only new operations
on documents will cause the versioning system to start and kick in for that
doc.
A scan type would be awesome. I can raise a ticket for that. This
would also remove the need for working around the need to determine
the size of the results (admitidly not a big issue unless under update
load where the counts will drift between the count call and then a
search.)
Are there other strategies people are using to verify their index
state is consistent with some external 'truth'?
The versioning is not stored as a field, but you get it back when you do a "get" or when searching (per hit). Regarding getting back all the hits for a query (can be match_all), you can use search, with scrolling. The problem is that with large result sets, it can get very expensive, even with scrolling. I am thinking of adding a search_type called "scan" that will support something like that, without the requirement to do any type of sorting (score or other fields).
This is great Shay, an optimistic locking pattern usage. I like it.
This reminded me of an old topic I had raised a while back: http://elasticsearch-users.115913.n3.nabble.com/Index-Verification-td1430219.html . We're beginning to use ES for some production support tools at the moment, but the real goal is to surgically remove the cancer that is our own home grown index infrastructure (going to take a while).
Relating to Versioning stuff, but is that field an automatically stored field? In an ideal case I'd love an API to hit to return all _id and _version values for all items in a given index, I can drop the sort need from the original post, as I can probably do that outside ES. I'm after the fastest stream of these 2 fields from an index to be compared with another stream coming from the originating data source (our db) to validate the info in ES.
I notice that this Versioning should address Issue # 490? (491 I agree should have the _version information provided by the client, allow that to worry about what clock to use, in our case, our DB is the 'primary source of truth' and it generates a trigger-based lastupdated timestamp for the row).
love your work (and the amount of it.. )
Paul
On 4 January 2011 13:07, Shay Banon shay.banon@elasticsearch.com wrote:
The versioning feature allows to handle conflicts when doing a get/search and then index/delete. Each document has a version number, and each index and delete (tombstone) increase the version number.
A conflict is detected when providing a version parameter to the index/delete request and there is a mismatch between the version provided and the real time version of the current document. If a version number is not provided, then the operation is forced without doing any version check.
Search and get operations return the version number associated with the hit under _version. Note, the version number reflects the near real time status of the search. In other words, the version number will point to the doc when the index was last refreshed.
Nice side affects of this feature include returning on delete if the document was actually found or not, as well as fixing a rare out of order replication sync between a primary and its replica (under heavy changes of the same doc within a short period of time).
The bulk operation supports providing a version number as well using _version on each bulk item.
Upgrade Notes-------------
The versioning support is backward compatible, though only new operations on documents will cause the versioning system to start and kick in for that doc.
Would you also need the total_hits in the response? There are basically two
ways to implement it:
Do it shard by shard, until exhausted. Once you start iterating on a
shard, it is guaranteed that the data won't change (point in time snapshot),
and then, when you head over to the next shard, it does the point in time
and continues from there. The downside to that is that the total_hits can't
be computed.
Do a broadcast and point in time all shards, and then do shard by shard.
This is more tricky to implement, and requires more resources than 1.
A scan type would be awesome. I can raise a ticket for that. This
would also remove the need for working around the need to determine
the size of the results (admitidly not a big issue unless under update
load where the counts will drift between the count call and then a
search.)
Are there other strategies people are using to verify their index
state is consistent with some external 'truth'?
The versioning is not stored as a field, but you get it back when you do
a "get" or when searching (per hit). Regarding getting back all the hits for
a query (can be match_all), you can use search, with scrolling. The problem
is that with large result sets, it can get very expensive, even with
scrolling. I am thinking of adding a search_type called "scan" that will
support something like that, without the requirement to do any type of
sorting (score or other fields).
This is great Shay, an optimistic locking pattern usage. I like it.
This reminded me of an old topic I had raised a while back: http://elasticsearch-users.115913.n3.nabble.com/Index-Verification-td1430219.html. We're beginning to use ES for some production support tools at the moment,
but the real goal is to surgically remove the cancer that is our own home
grown index infrastructure (going to take a while).
Relating to Versioning stuff, but is that field an automatically stored
field? In an ideal case I'd love an API to hit to return all _id and
_version values for all items in a given index, I can drop the sort need
from the original post, as I can probably do that outside ES. I'm after the fastest stream of these 2 fields from an index to be compared with another
stream coming from the originating data source (our db) to validate the info
in ES.
I notice that this Versioning should address Issue # 490? (491 I agree
should have the _version information provided by the client, allow that to
worry about what clock to use, in our case, our DB is the 'primary source of
truth' and it generates a trigger-based lastupdated timestamp for the row).
love your work (and the amount of it.. )
Paul
On 4 January 2011 13:07, Shay Banon shay.banon@elasticsearch.com
wrote:
The versioning feature allows to handle conflicts when doing a get/search
and then index/delete. Each document has a version number, and each index
and delete (tombstone) increase the version number.
A conflict is detected when providing a version parameter to the
index/delete request and there is a mismatch between the version provided
and the real time version of the current document. If a version number is
not provided, then the operation is forced without doing any version check.
Search and get operations return the version number associated with the
hit under _version. Note, the version number reflects the near real time
status of the search. In other words, the version number will point to the
doc when the index was last refreshed.
Nice side affects of this feature include returning on delete if the
document was actually found or not, as well as fixing a rare out of order
replication sync between a primary and its replica (under heavy changes of
the same doc within a short period of time).
The bulk operation supports providing a version number as well using _version on each bulk item.
Upgrade Notes-------------
The versioning support is backward compatible, though only new operations
on documents will cause the versioning system to start and kick in for that
doc.
Would you also need the total_hits in the response? There are basically two
ways to implement it:
Do it shard by shard, until exhausted. Once you start iterating on a
shard, it is guaranteed that the data won't change (point in time snapshot),
and then, when you head over to the next shard, it does the point in time
and continues from there. The downside to that is that the total_hits can't
be computed.
Do a broadcast and point in time all shards, and then do shard by shard.
This is more tricky to implement, and requires more resources than 1.
I think I can live with (1) easily, because the counts themselves are
'meaningless' when comparing with the 'truth'. It is possible that the same
count of items exist in the original source as in in the index, but that's
not really indicative of much really, only comparing the 'version'
(lastupdate timestamp/version whatever) of each item and computing a 'diff'
between the truth and the index gives 100% confidence.
There are 3 states we (and really I would have thought anyone) would love to
be able to determine. "Source" is the data source that holds the 'truth'
(usually a DB):
Which items are in the source, but not in the index (a missed insert, or
a botched delete, completely missing in the sequence, this is why sorting is
important, because it's a sorted stream comparison, but I think the
verification client should handle the sort to simplify it for ES)
Which items are in the index but not in the source (a missed delete for
the index, need to be purged)
Which items are in the index, but the version (timestamps/checksums) do
not match - these will need to be reindexed from the source to be accurate.
On the way in on the train this morning I thought of perhaps having the
verification client talk direct to each primary shard server to get a shard
stream (then it can be parallelized I guess), but if you can do (1) and
provide that stream from an API, the comparisons with the source stream will
then verify state.
that's nirvana. The rationale for having such a tool is to mitigate the
need to do a full reindex to guarantee clean state, when one has a very
large source to index, and time is critical, a regular index verification is
way quicker (for us, with our current index framework, an index verification
against a few hundred million records on a single index server takes about
40 minutes, whereas a reindex takes over a day).
This I would have thought is even more useful when combined with gateway
snapshots in a DR sort of case. The source (db) in the DR may be further
ahead (or behind if the DR db can't be rolled forward far enough) of the
gateway snapshot one has in a DR centre, so it's quicker in a DR failover to
do a verification than a full reindex.
Hope that makes sense. I've been trying to learn the internals of ES to
start hacking and contributing things back, such as some more fine grained
performance monitoring stuff to expose what's going on in the cluster.
How do you plan to do sorting on the client side if there is large result
set? Is it batch based sorting basically pulling 1k hits, sort check, and
repeat?
Btw, you can already get just specific fields when you search, if you
specify an empty array (or empty string in the URI) for fields, you will
just get back _type, _id, (and _version in master).
Would you also need the total_hits in the response? There are basically
two ways to implement it:
Do it shard by shard, until exhausted. Once you start iterating on a
shard, it is guaranteed that the data won't change (point in time snapshot),
and then, when you head over to the next shard, it does the point in time
and continues from there. The downside to that is that the total_hits can't
be computed.
Do a broadcast and point in time all shards, and then do shard by
shard. This is more tricky to implement, and requires more resources than 1.
I think I can live with (1) easily, because the counts themselves are
'meaningless' when comparing with the 'truth'. It is possible that the same
count of items exist in the original source as in in the index, but that's
not really indicative of much really, only comparing the 'version'
(lastupdate timestamp/version whatever) of each item and computing a 'diff'
between the truth and the index gives 100% confidence.
There are 3 states we (and really I would have thought anyone) would love
to be able to determine. "Source" is the data source that holds the 'truth'
(usually a DB):
Which items are in the source, but not in the index (a missed insert,
or a botched delete, completely missing in the sequence, this is why sorting
is important, because it's a sorted stream comparison, but I think the
verification client should handle the sort to simplify it for ES)
Which items are in the index but not in the source (a missed delete for
the index, need to be purged)
Which items are in the index, but the version (timestamps/checksums) do
not match - these will need to be reindexed from the source to be accurate.
On the way in on the train this morning I thought of perhaps having the
verification client talk direct to each primary shard server to get a shard
stream (then it can be parallelized I guess), but if you can do (1) and
provide that stream from an API, the comparisons with the source stream will
then verify state.
that's nirvana. The rationale for having such a tool is to mitigate the
need to do a full reindex to guarantee clean state, when one has a very
large source to index, and time is critical, a regular index verification is
way quicker (for us, with our current index framework, an index verification
against a few hundred million records on a single index server takes about
40 minutes, whereas a reindex takes over a day).
This I would have thought is even more useful when combined with gateway
snapshots in a DR sort of case. The source (db) in the DR may be further
ahead (or behind if the DR db can't be rolled forward far enough) of the
gateway snapshot one has in a DR centre, so it's quicker in a DR failover to
do a verification than a full reindex.
Hope that makes sense. I've been trying to learn the internals of ES to
start hacking and contributing things back, such as some more fine grained
performance monitoring stuff to expose what's going on in the cluster.
How do you plan to do sorting on the client side if there is large result
set? Is it batch based sorting basically pulling 1k hits, sort check, and
repeat?
I think probably my first cut I would take the stream and write it in a
fixed record format to disk, in our case we just need the _id (a long) and
the version ( a timestamp for us, so another long, so only 16 bytes per
record). Then MMap that file into memory and merge sort it by Id. Then the
stream coming from the DB, which is already in ID sorted order can be used
for comparison quickly. Absolute worst case if the MMap way doesn't work
(say memory problems), I could either rely on the existing DB for a temp
table for sorting, or run a local Derby inMemDb or something. There's
always /usr/bin/sort if I get desperate...
Btw, you can already get just specific fields when you search, if you
specify an empty array (or empty string in the URI) for fields, you will
just get back _type, _id, (and _version in master).
Yeah, I know the search api gets that, it's the predetermining the # hits to
get all, the fact that the result can change between those calls, and the
fact I'm wondering about what a very large resultset like that (with all the
sorting/scoring costs that might entail) will do to the ES service. A scan
would be awesome.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
)