# Possible collision with child document \_id generation? **URL:** \ **Category:** Elasticsearch\ **Created:** [July 25, 2011, 8:29pm UTC](https://discuss.elastic.co/t/possible-collision-with-child-document--id-generation/4948 "2011-07-25T20:29:58Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Dan\_Pilone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dan_pilone/32/3261_2.png) [@Dan\_Pilone](https://discuss.elastic.co/u/Dan_Pilone)\ **Post date:** [July 25, 2011, 8:29pm UTC](https://discuss.elastic.co/t/possible-collision-with-child-document--id-generation/4948/1 "2011-07-25T20:29:58Z")
All, We're indexing millions of documents into a 3 (sometimes 4) node elastic cluster with 3 shards. Starting with 0.17.1 (we didn't try 0.17.0) and continuing to today's source I'm getting an occasional exception bulk indexing. I'm seeing this: {"create": {"\_index":"echo","\_type":"measured\_parameter","\_id":"6RGi7lBOTd- H28o31svvfQ","error":"DocumentAlreadyExistsEngineException[[echo][2] [measured\_parameter][6RGi7lBOTd-H28o31svvfQ]: document already exists]"}} This is buried in with lots of successful creations. We have 10's of millions of documents already indexed and we're hitting the cluster with three separate "indexers" which are generating the JSON. These are child documents and I'm not sure how we're colliding on the IDs. How are IDs generated? Could they result in a collision? Is there something we can do to prevent it? Thanks -- Dan ---
**Author:** ![kikster](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kikster/32/3117_2.png) [@kikster](https://discuss.elastic.co/u/kikster)\ **Post date:** [July 25, 2011, 8:45pm UTC](https://discuss.elastic.co/t/possible-collision-with-child-document--id-generation/4948/2 "2011-07-25T20:45:52Z")
I'm sure you've already thought of this but...couldn't you just explicitly set the ID of the document you want to index (if the children already have unique ids associated with them)? curl -XPOST '[http://localhost:9200/echo/measured\_parameter/3](http://localhost:9200/echo/measured_parameter/3)' -d ....... ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 26, 2011, 5:51am UTC](https://discuss.elastic.co/t/possible-collision-with-child-document--id-generation/4948/3 "2011-07-26T05:51:50Z")
Heya, Strange..., the id is generated using version 4 128bit UUID (basically, Java UUID generation). There shouldn't be conflicts... . Is it something that you can reproduce each time? On a smaller scale? -shay.banon On Mon, Jul 25, 2011 at 11:29 PM, Dan Pilone [dan@element84.com](mailto:dan@element84.com) wrote: > All, > We're indexing millions of documents into a 3 (sometimes 4) node > elastic cluster with 3 shards. Starting with 0.17.1 (we didn't try > 0.17.0) and continuing to today's source I'm getting an occasional > exception bulk indexing. I'm seeing this: > > {"create": > {"\_index":"echo","\_type":"measured\_parameter","\_id":"6RGi7lBOTd- > H28o31svvfQ","error":"DocumentAlreadyExistsEngineException[[echo][2] > [measured\_parameter][6RGi7lBOTd-H28o31svvfQ]: document already > exists]"}} > > This is buried in with lots of successful creations. We have 10's of > millions of documents already indexed and we're hitting the cluster > with three separate "indexers" which are generating the JSON. These > are child documents and I'm not sure how we're colliding on the IDs. > How are IDs generated? Could they result in a collision? Is there > something we can do to prevent it? Thanks -- Dan ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [July 26, 2011, 8:20pm UTC](https://discuss.elastic.co/t/possible-collision-with-child-document--id-generation/4948/4 "2011-07-26T20:20:27Z")
Hiya > Strange..., the id is generated using version 4 128bit UUID > (basically, Java UUID generation). There shouldn't be conflicts... . > Is it something that you can reproduce each time? On a smaller scale? I haven't experienced this myself, but I've heard reports from two people while bulk re-indexing millions of records from one index to another. > The process suddenly dies with 'document already exists', but it can't already exist, because the only thing writing to the new index is the reindexing process. My gut feeling is that this is a timing issue, and the bulk indexer in ES up trying to index the same doc twice on one shard, by mistake. clint ---
**Author:** ![Dan\_Pilone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dan_pilone/32/3261_2.png) [@Dan\_Pilone](https://discuss.elastic.co/u/Dan_Pilone)\ **Post date:** [July 26, 2011, 10:48pm UTC](https://discuss.elastic.co/t/possible-collision-with-child-document--id-generation/4948/5 "2011-07-26T22:48:46Z")
I don't think I can reproduce this on a small scale. We're indexing 100's of millions of docs as fast as we can push them and we see it _maybe_ once an hour. I hadn't ever seen it before we really ratcheted up the rate we were indexing documents so I'm inclined to agree that it's a timing/race kind of scenario. Interestingly, I think we only see it with child documents, but that may be a red-herring. Right now our ratio of child to parent documents is about 40:1 so obviously we're far more likely to see it there regardless of whether it could happen to any document. -- Dan -- Dan Pilone Managing Partner, Element 84 LLC [www.element84.com](http://www.element84.com) / [dan@element84.com](mailto:dan@element84.com) / 703-622-7370 On Tue, Jul 26, 2011 at 4:20 PM, Clinton Gormley [clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk)wrote: > Hiya > > > Strange..., the id is generated using version 4 128bit UUID > > (basically, Java UUID generation). There shouldn't be conflicts... . > > Is it something that you can reproduce each time? On a smaller scale? > > I haven't experienced this myself, but I've heard reports from two > people while bulk re-indexing millions of records from one index to > another. > > > > > The process suddenly dies with 'document already exists', but it can't > already exist, because the only thing writing to the new index is the > reindexing process. > > My gut feeling is that this is a timing issue, and the bulk indexer in > ES up trying to index the same doc twice on one shard, by mistake. > > clint ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:59am UTC](https://discuss.elastic.co/t/possible-collision-with-child-document--id-generation/4948/6 "2017-07-06T03:59:12Z")