# "Index IF NO DUPLICATE" Operation
**URL:** https://discuss.elastic.co/t/index-if-no-duplicate-operation/11186
**Category:** Elasticsearch
**Created:** [March 18, 2013, 9:09am UTC](https://discuss.elastic.co/t/index-if-no-duplicate-operation/11186 "2013-03-18T09:09:00Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)
#### Post date: [March 18, 2013, 9:09am UTC](https://discuss.elastic.co/t/index-if-no-duplicate-operation/11186/1 "2013-03-18T09:09:00Z")
Hi ,
I want to index a feed but then i need to make sure there is no duplicates
for it.
Is this available as a atomic operation at ES side.
My application pumps feeds in parallel fashion to ES. So adding something
like this on the application side will choke its performance badly.
Thanks
Vineeth
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [March 18, 2013, 9:12am UTC](https://discuss.elastic.co/t/index-if-no-duplicate-operation/11186/2 "2013-03-18T09:12:28Z")
Hey
Maybe the op\_type parameter can help in your case (depends if you need the
most up-to-date record in ES)...
> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.
On Mon, Mar 18, 2013 at 10:09 AM, Vineeth Mohan
[vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote:
> Hi ,
>
> I want to index a feed but then i need to make sure there is no duplicates
> for it.
> Is this available as a atomic operation at ES side.
>
> My application pumps feeds in parallel fashion to ES. So adding something
> like this on the application side will choke its performance badly.
>
> Thanks
> Vineeth
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)
#### Post date: [March 18, 2013, 9:23am UTC](https://discuss.elastic.co/t/index-if-no-duplicate-operation/11186/3 "2013-03-18T09:23:07Z")
I don't have any notion if the document that needs to be "Update or Index"
nature exist.
If i need to use op\_type parameter , i need the ID to a potentially same
document right.
Which i don't have in any way.
Thanks
Vineeth
On Mon, Mar 18, 2013 at 2:42 PM, Alexander Reelsen [alr@spinscale.de](mailto:alr@spinscale.de) wrote:
> Hey
>
> Maybe the op\_type parameter can help in your case (depends if you need the
> most up-to-date record in ES)...
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/index_.html)
>
> On Mon, Mar 18, 2013 at 10:09 AM, Vineeth Mohan \<[vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)
>
> > wrote:
>
> > Hi ,
> >
> > I want to index a feed but then i need to make sure there is no
> > duplicates for it.
> > Is this available as a atomic operation at ES side.
> >
> > My application pumps feeds in parallel fashion to ES. So adding something
> > like this on the application side will choke its performance badly.
> >
> > Thanks
> >
> > ```
> > Vineeth
> >
> > ```
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)
#### Post date: [March 18, 2013, 9:29am UTC](https://discuss.elastic.co/t/index-if-no-duplicate-operation/11186/4 "2013-03-18T09:29:20Z")
On Mon, 2013-03-18 at 14:53 +0530, Vineeth Mohan wrote:
> I don't have any notion if the document that needs to be "Update or
> Index" nature exist.
> If i need to use op\_type parameter , i need the ID to a potentially
> same document right.
> Which i don't have in any way.
Given that the only unique key constraint is on the \_id, if you want to
detect clashes, then you need to put that info into the \_id.
In a Perl module I wrote, I used the \_ids from a separate index to apply
a unique constraint to a field other than \_id in my main index:
[https://metacpan.org/module/ElasticSearchX::UniqueKey](https://metacpan.org/module/ElasticSearchX::UniqueKey)
clint
>
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)
#### Post date: [March 18, 2013, 9:29am UTC](https://discuss.elastic.co/t/index-if-no-duplicate-operation/11186/5 "2013-03-18T09:29:32Z")
Here by duplicate , i dont mean duplicate docID but duplicate content in
certain fields.
Thanks
Vineeth
On Mon, Mar 18, 2013 at 2:53 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote:
> I don't have any notion if the document that needs to be "Update or Index"
> nature exist.
> If i need to use op\_type parameter , i need the ID to a potentially same
> document right.
> Which i don't have in any way.
>
> Thanks
> Vineeth
>
> On Mon, Mar 18, 2013 at 2:42 PM, Alexander Reelsen [alr@spinscale.de](mailto:alr@spinscale.de)wrote:
>
> > Hey
> >
> > Maybe the op\_type parameter can help in your case (depends if you need
> > the most up-to-date record in ES)...
> > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/index_.html)
> >
> > On Mon, Mar 18, 2013 at 10:09 AM, Vineeth Mohan \<
> > [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)\> wrote:
> >
> > > Hi ,
> > >
> > > I want to index a feed but then i need to make sure there is no
> > > duplicates for it.
> > > Is this available as a atomic operation at ES side.
> > >
> > > My application pumps feeds in parallel fashion to ES. So adding
> > > something like this on the application side will choke its performance
> > > badly.
> > >
> > > Thanks
> > >
> > > ```
> > > Vineeth
> > >
> > > ```
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)
#### Post date: [March 18, 2013, 9:38am UTC](https://discuss.elastic.co/t/index-if-no-duplicate-operation/11186/6 "2013-03-18T09:38:54Z")
So the solution you propose is to make the id from the data fields that i
want as unique and then create a feed using it ?
Thanks
Vineeth
On Mon, Mar 18, 2013 at 2:59 PM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com)wrote:
> On Mon, 2013-03-18 at 14:53 +0530, Vineeth Mohan wrote:
>
> > I don't have any notion if the document that needs to be "Update or
> > Index" nature exist.
> > If i need to use op\_type parameter , i need the ID to a potentially
> > same document right.
> > Which i don't have in any way.
>
> Given that the only unique key constraint is on the \_id, if you want to
> detect clashes, then you need to put that info into the \_id.
>
> In a Perl module I wrote, I used the \_ids from a separate index to apply
> a unique constraint to a field other than \_id in my main index:
>
> [https://metacpan.org/module/ElasticSearchX::UniqueKey](https://metacpan.org/module/ElasticSearchX::UniqueKey)
>
> clint
>
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [March 18, 2013, 9:39am UTC](https://discuss.elastic.co/t/index-if-no-duplicate-operation/11186/7 "2013-03-18T09:39:35Z")
You have to know the certain field beforehand, collect the values, and
generate a cryptographic key of it (e.g. SHA-256). The key can be
formatted into base64 / hex characters and used for \_id. In conjunction
with the "create" op\_type and/or external versioning, you can implement
a strategy where you get a response if the document already exists.
I use this in the JDBC river.
Jörg
Am 18.03.13 10:29, schrieb Vineeth Mohan:
> Here by duplicate , i dont mean duplicate docID but duplicate content
> in certain fields.
>
> Thanks
> Vineeth
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)
#### Post date: [March 18, 2013, 9:45am UTC](https://discuss.elastic.co/t/index-if-no-duplicate-operation/11186/8 "2013-03-18T09:45:34Z")
Looks good , but if tomorrow i have to change the unique key function (
Like if some new field has to be considered) it would be tedious.
Thanks
Vineeth
On Mon, Mar 18, 2013 at 3:09 PM, Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote:
> You have to know the certain field beforehand, collect the values, and
> generate a cryptographic key of it (e.g. SHA-256). The key can be formatted
> into base64 / hex characters and used for \_id. In conjunction with the
> "create" op\_type and/or external versioning, you can implement a strategy
> where you get a response if the document already exists.
>
> I use this in the JDBC river.
>
> Jörg
>
> Am 18.03.13 10:29, schrieb Vineeth Mohan:
>
> Here by duplicate , i dont mean duplicate docID but duplicate content in
>
> > certain fields.
> >
> > Thanks
> > Vineeth
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com)
> .
> For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> .
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)
#### Post date: [March 18, 2013, 12:09pm UTC](https://discuss.elastic.co/t/index-if-no-duplicate-operation/11186/9 "2013-03-18T12:09:37Z")
Again one more doubt.
Lets say if its a duplicate , i want to update the original document
(Increment the occurance field) , is it possible using this logic ?
Thanks
Vineeth
On Mon, Mar 18, 2013 at 3:15 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote:
> Looks good , but if tomorrow i have to change the unique key function (
> Like if some new field has to be considered) it would be tedious.
>
> Thanks
> Vineeth
>
> On Mon, Mar 18, 2013 at 3:09 PM, Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com)wrote:
>
> > You have to know the certain field beforehand, collect the values, and
> > generate a cryptographic key of it (e.g. SHA-256). The key can be formatted
> > into base64 / hex characters and used for \_id. In conjunction with the
> > "create" op\_type and/or external versioning, you can implement a strategy
> > where you get a response if the document already exists.
> >
> > I use this in the JDBC river.
> >
> > Jörg
> >
> > Am 18.03.13 10:29, schrieb Vineeth Mohan:
> >
> > Here by duplicate , i dont mean duplicate docID but duplicate content in
> >
> > > certain fields.
> > >
> > > Thanks
> > > Vineeth
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com)
> > .
> > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> > .
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@brian\_yoder](https://discuss.elastic.co/u/brian_yoder)
#### Post date: [March 18, 2013, 2:59pm UTC](https://discuss.elastic.co/t/index-if-no-duplicate-operation/11186/10 "2013-03-18T14:59:44Z")
The "create" action will create the record if the index+type+id does not
exist, but will reject it if the index+type+id already exists.
For example, a \_bulk action-and-meta-data line specifies create as follows:
{ "_create_" : { "\_index" : "fizzbuzz", "\_type" : "person", "\_id" :
"7214560012" } }
On Monday, March 18, 2013 5:09:00 AM UTC-4, Vineeth Mohan wrote:
> Hi ,
>
> I want to index a feed but then i need to make sure there is no duplicates
> for it.
> Is this available as a atomic operation at ES side.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)
#### Post date: [March 18, 2013, 3:07pm UTC](https://discuss.elastic.co/t/index-if-no-duplicate-operation/11186/11 "2013-03-18T15:07:37Z")
What if i want to modify the original feed based on the new feed instead of
rejecting the second feed ?
Thanks
Vineeth
On Mon, Mar 18, 2013 at 8:29 PM, InquiringMind [brian.from.fl@gmail.com](mailto:brian.from.fl@gmail.com)wrote:
> The "create" action will create the record if the index+type+id does not
> exist, but will reject it if the index+type+id already exists.
>
> For example, a \_bulk action-and-meta-data line specifies create as follows:
>
> { "_create_" : { "\_index" : "fizzbuzz", "\_type" : "person", "\_id" : "
> 7214560012" } }
>
> On Monday, March 18, 2013 5:09:00 AM UTC-4, Vineeth Mohan wrote:
>
> > Hi ,
> >
> > I want to index a feed but then i need to make sure there is no
> > duplicates for it.
> > Is this available as a atomic operation at ES side.
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@brian\_yoder](https://discuss.elastic.co/u/brian_yoder)
#### Post date: [March 18, 2013, 7:33pm UTC](https://discuss.elastic.co/t/index-if-no-duplicate-operation/11186/12 "2013-03-18T19:33:14Z")
Well, the version number from the original feed can be passed into the
update. That way, when you computed the updated version and went to write
it back, the version number would be checked and used to ensure that no
other updates were done to that record.
In case of failure due to version number mismatch, you could re-read the
record and re-compute the changes, and then try again.
I believe that would ensure an atomic update. There is no record
lock/read/update/unlock facility, of course. Which is a good thing!
On Monday, March 18, 2013 11:07:37 AM UTC-4, Vineeth Mohan wrote:
> What if i want to modify the original feed based on the new feed instead
> of rejecting the second feed ?
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 2:46am UTC](https://discuss.elastic.co/t/index-if-no-duplicate-operation/11186/13 "2017-07-06T02:46:00Z")