# Idempotent indexing into ElasticSearch when neither data nor version number have changed?
**URL:** https://discuss.elastic.co/t/idempotent-indexing-into-elasticsearch-when-neither-data-nor-version-number-have-changed/7715
**Category:** Elasticsearch
**Created:** [May 16, 2012, 8:39am UTC](https://discuss.elastic.co/t/idempotent-indexing-into-elasticsearch-when-neither-data-nor-version-number-have-changed/7715 "2012-05-16T08:39:17Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@Michael\_Snell](https://discuss.elastic.co/u/Michael_Snell)
#### Post date: [May 16, 2012, 8:39am UTC](https://discuss.elastic.co/t/idempotent-indexing-into-elasticsearch-when-neither-data-nor-version-number-have-changed/7715/1 "2012-05-16T08:39:17Z")
Hi - Currently, when using external versioning, ElasticSearch always
returns the same error when there is a version number conflict, eg:
VersionConflictEngineException[[twitter][2] [tweet][1]: version conflict,
current [3], required [2]]
Is it possible to change this so that a different error is returned when
neither the version number nor the data has changed? In other words, it
should be possible to differentiate between attempting to reindex the same
data with the same version number (which is inefficient but harmless) with
attempting to index different data with the same version number (which
indicates a bug version numbering in the source system).
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [May 16, 2012, 9:58pm UTC](https://discuss.elastic.co/t/idempotent-indexing-into-elasticsearch-when-neither-data-nor-version-number-have-changed/7715/2 "2012-05-16T21:58:35Z")
If you are using the Java API, then you can get the current version and
provided version in the VersionConflictEngineException, if you are using
the REST API, your best bet is to parse it. I have been meaning to allow
for "failures" to be serialized into json (for example) to provide more
metadata on the failure itself for the REST API.
On Wed, May 16, 2012 at 11:39 AM, Michael Snell [michael@snell.com](mailto:michael@snell.com) wrote:
> Hi - Currently, when using external versioning, Elasticsearch always
> returns the same error when there is a version number conflict, eg:
>
> VersionConflictEngineException[[twitter][2] [tweet][1]: version conflict,
> current [3], required [2]]
>
> Is it possible to change this so that a different error is returned when
> neither the version number nor the data has changed? In other words, it
> should be possible to differentiate between attempting to reindex the same
> data with the same version number (which is inefficient but harmless) with
> attempting to index different data with the same version number (which
> indicates a bug version numbering in the source system).
---
### Author:  [@Michael\_Snell](https://discuss.elastic.co/u/Michael_Snell)
#### Post date: [May 17, 2012, 7:52am UTC](https://discuss.elastic.co/t/idempotent-indexing-into-elasticsearch-when-neither-data-nor-version-number-have-changed/7715/3 "2012-05-17T07:52:21Z")
Hi - I don't think I explained the problem clearly: The issue is, given
that we can already tell the version number hasn't changed, can we tell
whether the data itself has changed or not?
In our case, we use the Java API and already parse out the current and
provided version number from the error string (using a regular expression)
so we know if we've attempted to index the same version. However, we need
to differentiate between:
1. Same version and same data: Inefficient but harmless, we'd like to
ignore this (could be caused for example by indexing a list of items which
include some which have not actually changed)
2. Same version but different data: This indicates a data inconsistency, so
we'd like to raise an exception if this occurs (could be caused for example
by attempting to index data which has been updated in the database via a
manual SQL update, but without the version column being incremented)
On Wednesday, 16 May 2012 22:58:35 UTC+1, kimchy wrote:
> If you are using the Java API, then you can get the current version and
> provided version in the VersionConflictEngineException, if you are using
> the REST API, your best bet is to parse it. I have been meaning to allow
> for "failures" to be serialized into json (for example) to provide more
> metadata on the failure itself for the REST API.
>
> On Wed, May 16, 2012 at 11:39 AM, Michael Snell [michael@snell.com](mailto:michael@snell.com) wrote:
>
> > Hi - Currently, when using external versioning, Elasticsearch always
> > returns the same error when there is a version number conflict, eg:
> >
> > VersionConflictEngineException[[twitter][2] [tweet][1]: version conflict,
> > current [3], required [2]]
> >
> > Is it possible to change this so that a different error is returned when
> > neither the version number nor the data has changed? In other words, it
> > should be possible to differentiate between attempting to reindex the same
> > data with the same version number (which is inefficient but harmless) with
> > attempting to index different data with the same version number (which
> > indicates a bug version numbering in the source system).
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [May 20, 2012, 7:41pm UTC](https://discuss.elastic.co/t/idempotent-indexing-into-elasticsearch-when-neither-data-nor-version-number-have-changed/7715/4 "2012-05-20T19:41:24Z")
Ahh, I see. Since the index response does not return the data indexed on
version conflict failure, then no, you can't. You could do a get before and
then possibly compare, and use the version returned from get to update the
data...
On Thu, May 17, 2012 at 9:52 AM, Michael Snell [michael@snell.com](mailto:michael@snell.com) wrote:
> Hi - I don't think I explained the problem clearly: The issue is, given
> that we can already tell the version number hasn't changed, can we tell
> whether the data itself has changed or not?
>
> In our case, we use the Java API and already parse out the current and
> provided version number from the error string (using a regular expression)
> so we know if we've attempted to index the same version. However, we need
> to differentiate between:
>
> 1. Same version and same data: Inefficient but harmless, we'd like to
> ignore this (could be caused for example by indexing a list of items which
> include some which have not actually changed)
>
> 2. Same version but different data: This indicates a data inconsistency,
> so we'd like to raise an exception if this occurs (could be caused for
> example by attempting to index data which has been updated in the database
> via a manual SQL update, but without the version column being incremented)
>
> On Wednesday, 16 May 2012 22:58:35 UTC+1, kimchy wrote:
>
> > If you are using the Java API, then you can get the current version and
> > provided version in the **VersionConflictEngineException** , if you are
> > using the REST API, your best bet is to parse it. I have been meaning to
> > allow for "failures" to be serialized into json (for example) to provide
> > more metadata on the failure itself for the REST API.
> >
> > On Wed, May 16, 2012 at 11:39 AM, Michael Snell [michael@snell.com](mailto:michael@snell.com)wrote:
> >
> > > Hi - Currently, when using external versioning, Elasticsearch always
> > > returns the same error when there is a version number conflict, eg:
> > >
> > > VersionConflictEngineException\*\*[[twitter][2] [tweet][1]: version
> > > conflict, current [3], required [2]]
> > >
> > > Is it possible to change this so that a different error is returned when
> > > neither the version number nor the data has changed? In other words, it
> > > should be possible to differentiate between attempting to reindex the same
> > > data with the same version number (which is inefficient but harmless) with
> > > attempting to index different data with the same version number (which
> > > indicates a bug version numbering in the source system).
---
### Author:  [@Michael\_Snell](https://discuss.elastic.co/u/Michael_Snell)
#### Post date: [May 22, 2012, 9:26pm UTC](https://discuss.elastic.co/t/idempotent-indexing-into-elasticsearch-when-neither-data-nor-version-number-have-changed/7715/5 "2012-05-22T21:26:05Z")
Hi - We've now implemented this ourselves, ie if a
VersionConflictEngineException is returned, use a regexp to extract the
current and provided version number. If they are equal, get the current
source in Elasticsearch and compare with the source we are trying to index.
Only raise an error if they differ.
This seems to work well enough, but I'm sure it would be more efficient if
Elasticsearch did this internally - perhaps a features request could be
raised for a future version?
On Sunday, 20 May 2012 20:41:24 UTC+1, kimchy wrote:
> Ahh, I see. Since the index response does not return the data indexed on
> version conflict failure, then no, you can't. You could do a get before and
> then possibly compare, and use the version returned from get to update the
> data...
>
> On Thu, May 17, 2012 at 9:52 AM, Michael Snell [michael@snell.com](mailto:michael@snell.com) wrote:
>
> > Hi - I don't think I explained the problem clearly: The issue is, given
> > that we can already tell the version number hasn't changed, can we tell
> > whether the data itself has changed or not?
> >
> > In our case, we use the Java API and already parse out the current and
> > provided version number from the error string (using a regular expression)
> > so we know if we've attempted to index the same version. However, we need
> > to differentiate between:
> >
> > 1. Same version and same data: Inefficient but harmless, we'd like to
> > ignore this (could be caused for example by indexing a list of items which
> > include some which have not actually changed)
> >
> > 2. Same version but different data: This indicates a data inconsistency,
> > so we'd like to raise an exception if this occurs (could be caused for
> > example by attempting to index data which has been updated in the database
> > via a manual SQL update, but without the version column being incremented)
> >
> > On Wednesday, 16 May 2012 22:58:35 UTC+1, kimchy wrote:
> >
> > > If you are using the Java API, then you can get the current version and
> > > provided version in the **VersionConflictEngineException** , if you are
> > > using the REST API, your best bet is to parse it. I have been meaning to
> > > allow for "failures" to be serialized into json (for example) to provide
> > > more metadata on the failure itself for the REST API.
> > >
> > > On Wed, May 16, 2012 at 11:39 AM, Michael Snell [michael@snell.com](mailto:michael@snell.com)wrote:
> > >
> > > > Hi - Currently, when using external versioning, Elasticsearch always
> > > > returns the same error when there is a version number conflict, eg:
> > > >
> > > > VersionConflictEngineException\*\*[[twitter][2] [tweet][1]: version
> > > > conflict, current [3], required [2]]
> > > >
> > > > Is it possible to change this so that a different error is returned
> > > > when neither the version number nor the data has changed? In other words,
> > > > it should be possible to differentiate between attempting to reindex the
> > > > same data with the same version number (which is inefficient but harmless)
> > > > with attempting to index different data with the same version number (which
> > > > indicates a bug version numbering in the source system).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 3:27am UTC](https://discuss.elastic.co/t/idempotent-indexing-into-elasticsearch-when-neither-data-nor-version-number-have-changed/7715/6 "2017-07-06T03:27:35Z")