# Just Pushed: Versioning
**URL:** https://discuss.elastic.co/t/just-pushed-versioning/3735
**Category:** Elasticsearch
**Created:** [January 4, 2011, 2:07am UTC](https://discuss.elastic.co/t/just-pushed-versioning/3735 "2011-01-04T02:07:06Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [January 4, 2011, 2:07am UTC](https://discuss.elastic.co/t/just-pushed-versioning/3735/1 "2011-01-04T02:07:06Z")
The versioning feature allows to handle conflicts when doing a get/search
and then index/delete. Each document has a version number, and each index
and delete (tombstone) increase the version number.
A conflict is detected when providing a version parameter to the
index/delete request and there is a mismatch between the version provided
and the _real time version_ of the current document. If a version number is
not provided, then the operation is forced without doing any version check.
Search and get operations return the version number associated with the hit
under `_version`. Note, the version number reflects the near real time
status of the search. In other words, the version number will point to the
doc when the index was last refreshed.
Nice side affects of this feature include returning on delete if the
document was actually found or not, as well as fixing a rare out of order
replication sync between a primary and its replica (under heavy changes of
the same doc within a short period of time).
The bulk operation supports providing a version number as well using
`_version` on each bulk item.
## Upgrade Notes
The versioning support is backward compatible, though only new operations on
documents will cause the versioning system to start and kick in for that
doc.
Issue: [https://github.com/elasticsearch/elasticsearch/issues/issue/594](https://github.com/elasticsearch/elasticsearch/issues/issue/594).
-shay.banon
---
### Author:  [@medcl\_net](https://discuss.elastic.co/u/medcl_net)
#### Post date: [January 4, 2011, 2:29am UTC](https://discuss.elastic.co/t/just-pushed-versioning/3735/2 "2011-01-04T02:29:15Z")
great feature,nice new year gift ,thanks for the hard working,i can see the time machine is coming~~
so cool!!
From: Shay Banon
Sent: Tuesday, January 04, 2011 10:07 AM
To: users
Subject: Just Pushed: Versioning
The versioning feature allows to handle conflicts when doing a get/search and then index/delete. Each document has a version number, and each index and delete (tombstone) increase the version number.
A conflict is detected when providing a version parameter to the index/delete request and there is a mismatch between the version provided and the _real time version_ of the current document. If a version number is not provided, then the operation is forced without doing any version check.
Search and get operations return the version number associated with the hit under `_version`. Note, the version number reflects the near real time status of the search. In other words, the version number will point to the doc when the index was last refreshed.
Nice side affects of this feature include returning on delete if the document was actually found or not, as well as fixing a rare out of order replication sync between a primary and its replica (under heavy changes of the same doc within a short period of time).
The bulk operation supports providing a version number as well using `_version` on each bulk item.
## Upgrade Notes
The versioning support is backward compatible, though only new operations on documents will cause the versioning system to start and kick in for that doc.
Issue: [https://github.com/elasticsearch/elasticsearch/issues/issue/594](https://github.com/elasticsearch/elasticsearch/issues/issue/594).
-shay.banon
---
### Author:  [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)
#### Post date: [January 4, 2011, 2:43am UTC](https://discuss.elastic.co/t/just-pushed-versioning/3735/3 "2011-01-04T02:43:45Z")
This is great Shay, an optimistic locking pattern usage. I like it.
This reminded me of an old topic I had raised a while back:
[http://elasticsearch-users.115913.n3.nabble.com/Index-Verification-td1430219.html](http://elasticsearch-users.115913.n3.nabble.com/Index-Verification-td1430219.html).
We're beginning to use ES for some production support tools at the
moment,
but the real goal is to surgically remove the cancer that is our own home
grown index infrastructure (going to take a while).
Relating to Versioning stuff, but is that field an automatically stored
field? In an ideal case I'd love an API to hit to return all \_id and
\_version values for all items in a given index, I can drop the sort need
from the original post, as I can probably do that outside ES. I'm after the
_fastest_ stream of these 2 fields from an index to be compared with another
stream coming from the originating data source (our db) to validate the info
in ES.
I notice that this Versioning should address Issue # 490? (491 I agree
should have the \_version information provided by the client, allow that to
worry about what clock to use, in our case, our DB is the 'primary source of
truth' and it generates a trigger-based lastupdated timestamp for the row).
love your work (and the amount of it.. 馃檪 )
Paul
On 4 January 2011 13:07, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> The versioning feature allows to handle conflicts when doing a get/search
> and then index/delete. Each document has a version number, and each index
> and delete (tombstone) increase the version number.
>
> A conflict is detected when providing a version parameter to the
> index/delete request and there is a mismatch between the version provided
> and the _real time version_ of the current document. If a version number is
> not provided, then the operation is forced without doing any version check.
>
> Search and get operations return the version number associated with the hit
> under `_version`. Note, the version number reflects the near real time
> status of the search. In other words, the version number will point to the
> doc when the index was last refreshed.
>
> Nice side affects of this feature include returning on delete if the
> document was actually found or not, as well as fixing a rare out of order
> replication sync between a primary and its replica (under heavy changes of
> the same doc within a short period of time).
>
> The bulk operation supports providing a version number as well using
> `_version` on each bulk item.
>
> ## Upgrade Notes
>
> The versioning support is backward compatible, though only new operations
> on documents will cause the versioning system to start and kick in for that
> doc.
>
> Issue: [Versioning 路 Issue #594 路 elastic/elasticsearch 路 GitHub](https://github.com/elasticsearch/elasticsearch/issues/issue/594).
> -shay.banon
---
### Author:  [@Mahendra\_M](https://discuss.elastic.co/u/Mahendra_M)
#### Post date: [January 5, 2011, 11:27am UTC](https://discuss.elastic.co/t/just-pushed-versioning/3735/4 "2011-01-05T11:27:04Z")
On Tue, Jan 4, 2011 at 7:37 AM, Shay Banon wrote:
> The versioning feature allows to handle conflicts when doing a get/search
> and then index/delete. Each document has a version number, and each index
> and delete (tombstone) increase the version number.
Hi,
This is a great feature. Even CouchDB has a '\_rev' field which it uses
to maintain versions between documents.
Do you think it would be a good idea to map the CouchDB '\_rev' field
to the ES '\_version' field ? Specially in the 'CouchDB river' plugin.
If so, I can do that.
PS: The problem would be solved if ES used '\_rev' 馃檪
Regards,
Mahendra
[http://twitter.com/mahendra](http://twitter.com/mahendra)
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [January 5, 2011, 11:43am UTC](https://discuss.elastic.co/t/just-pushed-versioning/3735/5 "2011-01-05T11:43:40Z")
The versioning is not stored as a field, but you get it back when you do a
"get" or when searching (per hit). Regarding getting back all the hits for a
query (can be match\_all), you can use search, with scrolling. The problem is
that with large result sets, it can get very expensive, even with scrolling.
I am thinking of adding a search\_type called "scan" that will support
something like that, without the requirement to do any type of sorting
(score or other fields).
On Tue, Jan 4, 2011 at 4:43 AM, Paul Smith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) wrote:
> This is great Shay, an optimistic locking pattern usage. I like it.
>
> This reminded me of an old topic I had raised a while back:
> [http://elasticsearch-users.115913.n3.nabble.com/Index-Verification-td1430219.html](http://elasticsearch-users.115913.n3.nabble.com/Index-Verification-td1430219.html). We're beginning to use ES for some production support tools at the moment,
> but the real goal is to surgically remove the cancer that is our own home
> grown index infrastructure (going to take a while).
>
> Relating to Versioning stuff, but is that field an automatically stored
> field? In an ideal case I'd love an API to hit to return all \_id and
> \_version values for all items in a given index, I can drop the sort need
> from the original post, as I can probably do that outside ES. I'm after the
> _fastest_ stream of these 2 fields from an index to be compared with another
> stream coming from the originating data source (our db) to validate the info
> in ES.
>
> I notice that this Versioning should address Issue # 490? (491 I agree
> should have the \_version information provided by the client, allow that to
> worry about what clock to use, in our case, our DB is the 'primary source of
> truth' and it generates a trigger-based lastupdated timestamp for the row).
>
> love your work (and the amount of it.. 馃檪 )
>
> Paul
>
> On 4 January 2011 13:07, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
>
> > The versioning feature allows to handle conflicts when doing a get/search
> > and then index/delete. Each document has a version number, and each index
> > and delete (tombstone) increase the version number.
> >
> > A conflict is detected when providing a version parameter to the
> > index/delete request and there is a mismatch between the version provided
> > and the _real time version_ of the current document. If a version number is
> > not provided, then the operation is forced without doing any version check.
> >
> > Search and get operations return the version number associated with the
> > hit under `_version`. Note, the version number reflects the near real time
> > status of the search. In other words, the version number will point to the
> > doc when the index was last refreshed.
> >
> > Nice side affects of this feature include returning on delete if the
> > document was actually found or not, as well as fixing a rare out of order
> > replication sync between a primary and its replica (under heavy changes of
> > the same doc within a short period of time).
> >
> > The bulk operation supports providing a version number as well using
> > `_version` on each bulk item.
> >
> > ## Upgrade Notes
> >
> > The versioning support is backward compatible, though only new operations
> > on documents will cause the versioning system to start and kick in for that
> > doc.
> >
> > Issue: [Versioning 路 Issue #594 路 elastic/elasticsearch 路 GitHub](https://github.com/elasticsearch/elasticsearch/issues/issue/594).
> > -shay.banon
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [January 5, 2011, 11:46am UTC](https://discuss.elastic.co/t/just-pushed-versioning/3735/6 "2011-01-05T11:46:10Z")
You can't use that since the version is internal to ES, and you can't index
a doc and control its version (if you provide the version, it will fail
since there is no such version, or it does not exists, the latter, to handle
deletes).
I don't think that you need to have that mapping with the couchdb
integration anyhow. Couchdb will maintain the correct stream of changes and
it will be applied to ES. Maybe you are after indexing the \_rev as an
addition field in the json, so you can get it back when you search? If thats
the case, I think we can enhance the couchdb river to support that.
On Wed, Jan 5, 2011 at 1:27 PM, Mahendra M [mahendra.m@gmail.com](mailto:mahendra.m@gmail.com) wrote:
> On Tue, Jan 4, 2011 at 7:37 AM, Shay Banon wrote:
>
> > The versioning feature allows to handle conflicts when doing a get/search
> > and then index/delete. Each document has a version number, and each index
> > and delete (tombstone) increase the version number.
>
> Hi,
>
> This is a great feature. Even CouchDB has a '\_rev' field which it uses
> to maintain versions between documents.
>
> Do you think it would be a good idea to map the CouchDB '\_rev' field
> to the ES '\_version' field ? Specially in the 'CouchDB river' plugin.
> If so, I can do that.
>
> PS: The problem would be solved if ES used '\_rev' 馃檪
>
> Regards,
> Mahendra
>
> [http://twitter.com/mahendra](http://twitter.com/mahendra)
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [January 5, 2011, 12:23pm UTC](https://discuss.elastic.co/t/just-pushed-versioning/3735/7 "2011-01-05T12:23:32Z")
Two more benefits of versioning that I forgot to mention:
1. Index with create flag will now fail if there is already an indexed doc.
This basically allows to use create as a putIfAbsent logic.
2. Indexing a document that does not exists will be as fast as with create
flag set to true. So no need to set it anymore to improve fresh data
indexing performance.
On Tue, Jan 4, 2011 at 4:07 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:
> The versioning feature allows to handle conflicts when doing a get/search
> and then index/delete. Each document has a version number, and each index
> and delete (tombstone) increase the version number.
>
> A conflict is detected when providing a version parameter to the
> index/delete request and there is a mismatch between the version provided
> and the _real time version_ of the current document. If a version number is
> not provided, then the operation is forced without doing any version check.
>
> Search and get operations return the version number associated with the hit
> under `_version`. Note, the version number reflects the near real time
> status of the search. In other words, the version number will point to the
> doc when the index was last refreshed.
>
> Nice side affects of this feature include returning on delete if the
> document was actually found or not, as well as fixing a rare out of order
> replication sync between a primary and its replica (under heavy changes of
> the same doc within a short period of time).
>
> The bulk operation supports providing a version number as well using
> `_version` on each bulk item.
>
> ## Upgrade Notes
>
> The versioning support is backward compatible, though only new operations
> on documents will cause the versioning system to start and kick in for that
> doc.
>
> Issue: [Versioning 路 Issue #594 路 elastic/elasticsearch 路 GitHub](https://github.com/elasticsearch/elasticsearch/issues/issue/594).
> -shay.banon
---
### Author:  [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)
#### Post date: [January 5, 2011, 9:20pm UTC](https://discuss.elastic.co/t/just-pushed-versioning/3735/8 "2011-01-05T21:20:19Z")
A scan type would be awesome. I can raise a ticket for that. This
would also remove the need for working around the need to determine
the size of the results (admitidly not a big issue unless under update
load where the counts will drift between the count call and then a
search.)
Are there other strategies people are using to verify their index
state is consistent with some external 'truth'?
On Wednesday, 5 January 2011, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> The versioning is not stored as a field, but you get it back when you do a "get" or when searching (per hit). Regarding getting back all the hits for a query (can be match\_all), you can use search, with scrolling. The problem is that with large result sets, it can get very expensive, even with scrolling. I am thinking of adding a search\_type called "scan" that will support something like that, without the requirement to do any type of sorting (score or other fields).
>
> On Tue, Jan 4, 2011 at 4:43 AM, Paul Smith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) wrote:
>
> This is great Shay, an optimistic locking pattern usage. I like it.
> This reminded me of an old topic I had raised a while back: [http://elasticsearch-users.115913.n3.nabble.com/Index-Verification-td1430219.html](http://elasticsearch-users.115913.n3.nabble.com/Index-Verification-td1430219.html) . We're beginning to use ES for some production support tools at the moment, but the real goal is to surgically remove the cancer that is our own home grown index infrastructure (going to take a while).
>
> Relating to Versioning stuff, but is that field an automatically stored field? In an ideal case I'd love an API to hit to return all \_id and \_version values for all items in a given index, I can drop the sort need from the original post, as I can probably do that outside ES. I'm after the _fastest_ stream of these 2 fields from an index to be compared with another stream coming from the originating data source (our db) to validate the info in ES.
>
> I notice that this Versioning should address Issue # 490? (491 I agree should have the \_version information provided by the client, allow that to worry about what clock to use, in our case, our DB is the 'primary source of truth' and it generates a trigger-based lastupdated timestamp for the row).
>
> love your work (and the amount of it.. 馃檪 )
> Paul
>
> On 4 January 2011 13:07, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> The versioning feature allows to handle conflicts when doing a get/search and then index/delete. Each document has a version number, and each index and delete (tombstone) increase the version number.
>
> A conflict is detected when providing a version parameter to the index/delete request and there is a mismatch between the version provided and the _real time version_ of the current document. If a version number is not provided, then the operation is forced without doing any version check.
>
> Search and get operations return the version number associated with the hit under `_version`. Note, the version number reflects the near real time status of the search. In other words, the version number will point to the doc when the index was last refreshed.
>
> Nice side affects of this feature include returning on delete if the document was actually found or not, as well as fixing a rare out of order replication sync between a primary and its replica (under heavy changes of the same doc within a short period of time).
>
> The bulk operation supports providing a version number as well using `_version` on each bulk item.
> Upgrade Notes-------------
> The versioning support is backward compatible, though only new operations on documents will cause the versioning system to start and kick in for that doc.
>
> Issue: [https://github.com/elasticsearch/elasticsearch/issues/issue/594.-shay.banon](https://github.com/elasticsearch/elasticsearch/issues/issue/594.-shay.banon)
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [January 5, 2011, 9:45pm UTC](https://discuss.elastic.co/t/just-pushed-versioning/3735/9 "2011-01-05T21:45:08Z")
Would you also need the total\_hits in the response? There are basically two
ways to implement it:
1. Do it shard by shard, until exhausted. Once you start iterating on a
shard, it is guaranteed that the data won't change (point in time snapshot),
and then, when you head over to the next shard, it does the point in time
and continues from there. The downside to that is that the total\_hits can't
be computed.
2. Do a broadcast and point in time all shards, and then do shard by shard.
This is more tricky to implement, and requires more resources than 1.
On Wed, Jan 5, 2011 at 11:20 PM, Paul Smith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) wrote:
> A scan type would be awesome. I can raise a ticket for that. This
> would also remove the need for working around the need to determine
> the size of the results (admitidly not a big issue unless under update
> load where the counts will drift between the count call and then a
> search.)
>
> Are there other strategies people are using to verify their index
> state is consistent with some external 'truth'?
>
> On Wednesday, 5 January 2011, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)
> wrote:
>
> > The versioning is not stored as a field, but you get it back when you do
> > a "get" or when searching (per hit). Regarding getting back all the hits for
> > a query (can be match\_all), you can use search, with scrolling. The problem
> > is that with large result sets, it can get very expensive, even with
> > scrolling. I am thinking of adding a search\_type called "scan" that will
> > support something like that, without the requirement to do any type of
> > sorting (score or other fields).
> >
> > On Tue, Jan 4, 2011 at 4:43 AM, Paul Smith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) wrote:
> >
> > This is great Shay, an optimistic locking pattern usage. I like it.
> > This reminded me of an old topic I had raised a while back:
> > [http://elasticsearch-users.115913.n3.nabble.com/Index-Verification-td1430219.html](http://elasticsearch-users.115913.n3.nabble.com/Index-Verification-td1430219.html). We're beginning to use ES for some production support tools at the moment,
> > but the real goal is to surgically remove the cancer that is our own home
> > grown index infrastructure (going to take a while).
> >
> > Relating to Versioning stuff, but is that field an automatically stored
> > field? In an ideal case I'd love an API to hit to return all \_id and
> > \_version values for all items in a given index, I can drop the sort need
> > from the original post, as I can probably do that outside ES. I'm after the
> > _fastest_ stream of these 2 fields from an index to be compared with another
> > stream coming from the originating data source (our db) to validate the info
> > in ES.
> >
> > I notice that this Versioning should address Issue # 490? (491 I agree
> > should have the \_version information provided by the client, allow that to
> > worry about what clock to use, in our case, our DB is the 'primary source of
> > truth' and it generates a trigger-based lastupdated timestamp for the row).
> >
> > love your work (and the amount of it.. 馃檪 )
> > Paul
> >
> > On 4 January 2011 13:07, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)
> > wrote:
> > The versioning feature allows to handle conflicts when doing a get/search
> > and then index/delete. Each document has a version number, and each index
> > and delete (tombstone) increase the version number.
> >
> > A conflict is detected when providing a version parameter to the
> > index/delete request and there is a mismatch between the version provided
> > and the _real time version_ of the current document. If a version number is
> > not provided, then the operation is forced without doing any version check.
> >
> > Search and get operations return the version number associated with the
> > hit under `_version`. Note, the version number reflects the near real time
> > status of the search. In other words, the version number will point to the
> > doc when the index was last refreshed.
> >
> > Nice side affects of this feature include returning on delete if the
> > document was actually found or not, as well as fixing a rare out of order
> > replication sync between a primary and its replica (under heavy changes of
> > the same doc within a short period of time).
> >
> > The bulk operation supports providing a version number as well using
> > `_version` on each bulk item.
> > Upgrade Notes-------------
> > The versioning support is backward compatible, though only new operations
> > on documents will cause the versioning system to start and kick in for that
> > doc.
> >
> > Issue:
> > [https://github.com/elasticsearch/elasticsearch/issues/issue/594.-shay.banon](https://github.com/elasticsearch/elasticsearch/issues/issue/594.-shay.banon)
---
### Author:  [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)
#### Post date: [January 5, 2011, 9:58pm UTC](https://discuss.elastic.co/t/just-pushed-versioning/3735/10 "2011-01-05T21:58:19Z")
On 6 January 2011 08:45, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> Would you also need the total\_hits in the response? There are basically two
> ways to implement it:
>
> 1. Do it shard by shard, until exhausted. Once you start iterating on a
> shard, it is guaranteed that the data won't change (point in time snapshot),
> and then, when you head over to the next shard, it does the point in time
> and continues from there. The downside to that is that the total\_hits can't
> be computed.
>
> 2. Do a broadcast and point in time all shards, and then do shard by shard.
> This is more tricky to implement, and requires more resources than 1.
I think I can live with (1) easily, because the counts themselves are
'meaningless' when comparing with the 'truth'. It is possible that the same
count of items exist in the original source as in in the index, but that's
not really indicative of much really, only comparing the 'version'
(lastupdate timestamp/version whatever) of each item and computing a 'diff'
between the truth and the index gives 100% confidence.
There are 3 states we (and really I would have thought anyone) would love to
be able to determine. "Source" is the data source that holds the 'truth'
(usually a DB):
- Which items are in the source, but _not_ in the index (a missed insert, or
a botched delete, completely missing in the sequence, this is why sorting is
important, because it's a sorted stream comparison, but I think the
verification client should handle the sort to simplify it for ES)
- Which items are in the index but _not_ in the source (a missed delete for
the index, need to be purged)
- Which items are in the index, but the version (timestamps/checksums) do
not match - these will need to be reindexed from the source to be accurate.
On the way in on the train this morning I thought of perhaps having the
verification client talk direct to each primary shard server to get a shard
stream (then it can be parallelized I guess), but if you can do (1) and
provide that stream from an API, the comparisons with the source stream will
then verify state.
that's nirvana. The rationale for having such a tool is to mitigate the
need to do a full reindex to guarantee clean state, when one has a very
large source to index, and time is critical, a regular index verification is
way quicker (for us, with our current index framework, an index verification
against a few hundred million records on a single index server takes about
40 minutes, whereas a reindex takes over a day).
This I would have thought is even more useful when combined with gateway
snapshots in a DR sort of case. The source (db) in the DR may be further
ahead (or behind if the DR db can't be rolled forward far enough) of the
gateway snapshot one has in a DR centre, so it's quicker in a DR failover to
do a verification than a full reindex.
Hope that makes sense. I've been trying to learn the internals of ES to
start hacking and contributing things back, such as some more fine grained
performance monitoring stuff to expose what's going on in the cluster.
cheers,
Paul
---
### Author:  [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)
#### Post date: [January 5, 2011, 10:13pm UTC](https://discuss.elastic.co/t/just-pushed-versioning/3735/11 "2011-01-05T22:13:37Z")
I just put up Issue
[https://github.com/elasticsearch/elasticsearch/issues/#issue/605](https://github.com/elasticsearch/elasticsearch/issues/#issue/605) to track
this.
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [January 5, 2011, 10:16pm UTC](https://discuss.elastic.co/t/just-pushed-versioning/3735/12 "2011-01-05T22:16:37Z")
How do you plan to do sorting on the client side if there is large result
set? Is it batch based sorting basically pulling 1k hits, sort check, and
repeat?
Btw, you can already get just specific fields when you search, if you
specify an empty array (or empty string in the URI) for fields, you will
just get back \_type, \_id, (and \_version in master).
On Wed, Jan 5, 2011 at 11:58 PM, Paul Smith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) wrote:
> On 6 January 2011 08:45, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
>
> > Would you also need the total\_hits in the response? There are basically
> > two ways to implement it:
> >
> > 1. Do it shard by shard, until exhausted. Once you start iterating on a
> > shard, it is guaranteed that the data won't change (point in time snapshot),
> > and then, when you head over to the next shard, it does the point in time
> > and continues from there. The downside to that is that the total\_hits can't
> > be computed.
> >
> > 2. Do a broadcast and point in time all shards, and then do shard by
> > shard. This is more tricky to implement, and requires more resources than 1.
>
> I think I can live with (1) easily, because the counts themselves are
> 'meaningless' when comparing with the 'truth'. It is possible that the same
> count of items exist in the original source as in in the index, but that's
> not really indicative of much really, only comparing the 'version'
> (lastupdate timestamp/version whatever) of each item and computing a 'diff'
> between the truth and the index gives 100% confidence.
>
> There are 3 states we (and really I would have thought anyone) would love
> to be able to determine. "Source" is the data source that holds the 'truth'
> (usually a DB):
>
> - Which items are in the source, but _not_ in the index (a missed insert,
> or a botched delete, completely missing in the sequence, this is why sorting
> is important, because it's a sorted stream comparison, but I think the
> verification client should handle the sort to simplify it for ES)
>
> - Which items are in the index but _not_ in the source (a missed delete for
> the index, need to be purged)
>
> - Which items are in the index, but the version (timestamps/checksums) do
> not match - these will need to be reindexed from the source to be accurate.
>
> On the way in on the train this morning I thought of perhaps having the
> verification client talk direct to each primary shard server to get a shard
> stream (then it can be parallelized I guess), but if you can do (1) and
> provide that stream from an API, the comparisons with the source stream will
> then verify state.
>
> that's nirvana. The rationale for having such a tool is to mitigate the
> need to do a full reindex to guarantee clean state, when one has a very
> large source to index, and time is critical, a regular index verification is
> way quicker (for us, with our current index framework, an index verification
> against a few hundred million records on a single index server takes about
> 40 minutes, whereas a reindex takes over a day).
>
> This I would have thought is even more useful when combined with gateway
> snapshots in a DR sort of case. The source (db) in the DR may be further
> ahead (or behind if the DR db can't be rolled forward far enough) of the
> gateway snapshot one has in a DR centre, so it's quicker in a DR failover to
> do a verification than a full reindex.
>
> Hope that makes sense. I've been trying to learn the internals of ES to
> start hacking and contributing things back, such as some more fine grained
> performance monitoring stuff to expose what's going on in the cluster.
>
> cheers,
>
> Paul
---
### Author:  [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)
#### Post date: [January 5, 2011, 10:24pm UTC](https://discuss.elastic.co/t/just-pushed-versioning/3735/13 "2011-01-05T22:24:04Z")
On 6 January 2011 09:16, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> How do you plan to do sorting on the client side if there is large result
> set? Is it batch based sorting basically pulling 1k hits, sort check, and
> repeat?
I think probably my first cut I would take the stream and write it in a
fixed record format to disk, in our case we just need the \_id (a long) and
the version ( a timestamp for us, so another long, so only 16 bytes per
record). Then MMap that file into memory and merge sort it by Id. Then the
stream coming from the DB, which is already in ID sorted order can be used
for comparison quickly. Absolute worst case if the MMap way doesn't work
(say memory problems), I could either rely on the existing DB for a temp
table for sorting, or run a local Derby inMemDb or something. There's
always /usr/bin/sort if I get desperate...
> Btw, you can already get just specific fields when you search, if you
> specify an empty array (or empty string in the URI) for fields, you will
> just get back \_type, \_id, (and \_version in master).
Yeah, I know the search api gets that, it's the predetermining the # hits to
get all, the fact that the result can change between those calls, and the
fact I'm wondering about what a very large resultset like that (with all the
sorting/scoring costs that might entail) will do to the ES service. A scan
would be awesome.
Paul
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 4:14am UTC](https://discuss.elastic.co/t/just-pushed-versioning/3735/14 "2017-07-06T04:14:14Z")