# Reliable term search **URL:** \ **Category:** Elasticsearch\ **Created:** [July 18, 2012, 9:11pm UTC](https://discuss.elastic.co/t/reliable-term-search/8456 "2012-07-18T21:11:47Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Eugene\_Strokin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eugene_strokin/32/1238_2.png) [@Eugene\_Strokin](https://discuss.elastic.co/u/Eugene_Strokin)\ **Post date:** [July 18, 2012, 9:11pm UTC](https://discuss.elastic.co/t/reliable-term-search/8456/1 "2012-07-18T21:11:47Z")
Hello, I have a problem, which I cannot reproduce locally nor on my DEV environment, but it is happening on production server. So, this is kind of mystery, but if someone could have any idea or any suggestion, I'd greatly appreciate it. One field of my document is not analyzed string, except it is lower cased. I need to keep this field unique, but it is not any ID or anything like this, just a field. So, when I need to index a new document I check if any document with such value in this field exists by just running term search with the new value. It works just fine everywhere. But in production (which is very busy some times), I've found documents with the same value. I've checked time when it was indexed, there are days and hours between those documents. I've added check if all nodes was searched and no errors during that Term Search, and if so, I don't index a document. But still I see from time to time that this term search doesn't return a document with the value, and the system inserts a new document with same value. I guess, my question: is the term search reliable? Meaning I will always get a document with the term I'm searching, and there are no errors indicated in result, or it is possible that search doesn't find the document for some reason. Or how to make sure, that my field always unique? Thanks in advance, Eugene S. ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [July 19, 2012, 8:26am UTC](https://discuss.elastic.co/t/reliable-term-search/8456/2 "2012-07-19T08:26:59Z")
Hi Eugene > One field of my document is not analyzed string, except it is lower > cased. I need to keep this field unique, but it is not any ID or > anything like this, just a field. > So, when I need to index a new document I check if any document with > such value in this field exists by just running term search with the > new value. > It works just fine everywhere. But in production (which is very busy > some times), I've found documents with the same value. I've checked > time when it was indexed, there are days and hours between those > documents. I've added check if all nodes was searched and no errors > during that Term Search, and if so, I don't index a document. But > still I see from time to time that this term search doesn't return a > document with the value, and the system inserts a new document with > same value. Note: search is near-real time. By default, search's "view" on the indexed data is refreshed once every second. So it is quite possible to have a document which has been indexed, (and which you can GET) but is not visible to search. I don't know what refresh interval you have set, but it seems unlikely that these docs were indexed hours or days before. A term search IS reliable (although it is possible that you have some other bug in your code which is interfering with that). Either way, your approach for managing a unique field is incorrect. You will always be subject to race conditions. An approach you can use is similar to what I used in [http://blogs.perl.org/users/clinton\_gormley/2011/10/elasticsearchsequence---a-blazing-fast-ticket-server.html](http://blogs.perl.org/users/clinton_gormley/2011/10/elasticsearchsequence---a-blazing-fast-ticket-server.html) The only unique field in ES is the \_id. So you can have an index whose job it is to maintain a list of unique values, stored in the \_id field. Eg, let's say you want to make sure that, for field 'my\_val', the value 'foo' is not used elsewhere. You can have an index 'unique', with a type 'my\_val'. Try to create a document as: - index: unique - type: myval - id: foo if it fails, then the value already exists (Of course, if the doc where you use that value is later deleted, then you need to delete the unique doc as well) clint ---
**Author:** ![Eugene\_Strokin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eugene_strokin/32/1238_2.png) [@Eugene\_Strokin](https://discuss.elastic.co/u/Eugene_Strokin)\ **Post date:** [July 20, 2012, 3:03am UTC](https://discuss.elastic.co/t/reliable-term-search/8456/3 "2012-07-20T03:03:51Z")
Thank you Clint, very good advise about separate index for unique values. I'll try to use it, but before, I'm trying to find a bug in my code, as you suggested, and the only possible thing I suspect is following: To make sure I'm receiving complete search result I'm checking response.getFailedShards()==0 But I also found response.getSuccessfulShards() I was thinking that response.getFailedShards()+response.getSuccessfulShards()=response.getTotalShards() but now I'm not sure. May be instead of checking response.getFailedShards()==0 I should be checking response.getSuccessfulShards()==response.getTotalShards(). Could someone please clarify? Thank you, Eugene S. On Thursday, July 19, 2012 4:26:59 AM UTC-4, Clinton Gormley wrote: > Hi Eugene > > > One field of my document is not analyzed string, except it is lower > > cased. I need to keep this field unique, but it is not any ID or > > anything like this, just a field. > > So, when I need to index a new document I check if any document with > > such value in this field exists by just running term search with the > > new value. > > It works just fine everywhere. But in production (which is very busy > > some times), I've found documents with the same value. I've checked > > time when it was indexed, there are days and hours between those > > documents. I've added check if all nodes was searched and no errors > > during that Term Search, and if so, I don't index a document. But > > still I see from time to time that this term search doesn't return a > > document with the value, and the system inserts a new document with > > same value. > > Note: search is near-real time. By default, search's "view" on the > indexed data is refreshed once every second. So it is quite possible to > have a document which has been indexed, (and which you can GET) but is > not visible to search. > > I don't know what refresh interval you have set, but it seems unlikely > that these docs were indexed hours or days before. A term search IS > reliable (although it is possible that you have some other bug in your > code which is interfering with that). > > Either way, your approach for managing a unique field is incorrect. You > will always be subject to race conditions. > > An approach you can use is similar to what I used in > > [ElasticSearch::Sequence - a blazing fast ticket server | Clinton Gormley [blogs.perl.org]](http://blogs.perl.org/users/clinton_gormley/2011/10/elasticsearchsequence---a-blazing-fast-ticket-server.html) > > The only unique field in ES is the \_id. > > So you can have an index whose job it is to maintain a list of unique > values, stored in the \_id field. > > Eg, let's say you want to make sure that, for field 'my\_val', the value > 'foo' is not used elsewhere. You can have an index 'unique', with a type > 'my\_val'. > > Try to create a document as: > > - index: unique > - type: myval > - id: foo > > if it fails, then the value already exists > > (Of course, if the doc where you use that value is later deleted, then > you need to delete the unique doc as well) > > clint ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [July 20, 2012, 8:12am UTC](https://discuss.elastic.co/t/reliable-term-search/8456/4 "2012-07-20T08:12:33Z")
Hiya > I was thinking that > response.getFailedShards()+response.getSuccessfulShards()=response.getTotalShards() but now I'm not sure. It should. But really you should only get failed shards in exceptional circumstances, eg a node is down. clint ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:19am UTC](https://discuss.elastic.co/t/reliable-term-search/8456/5 "2017-07-06T03:19:41Z")
Closed on July 6, 2017, 3:19am UTC