# Doc ID generation before communicating with ES **URL:** \ **Category:** Elasticsearch\ **Created:** [November 1, 2011, 12:57pm UTC](https://discuss.elastic.co/t/doc-id-generation-before-communicating-with-es/5736 "2011-11-01T12:57:57Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [November 1, 2011, 12:57pm UTC](https://discuss.elastic.co/t/doc-id-generation-before-communicating-with-es/5736/1 "2011-11-01T12:57:57Z")
Hi , I have a system which picks feed , does XYZ data transformations and then drops to ES. For each feed there is a feed ID associated. And when the feed reaches ES , we will receive a related docID. The feed ID and the doc ID are mapped somewhere to trace the path feed was picked , transformed and pushed to storage. I was wondering if there is some mechanism where a doc ID is generated at a point long before it reaches ES and i just have to maintain a single ID for the entire feed. So basically i should be able to generate a doc ID much before i push the feed to elastic Search and i should be able to guarantee that when the feed reaches the ES , that ID is free (as in no document is present with that ID). Is this possible with ES ? Thanks Vineeth ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [November 1, 2011, 1:05pm UTC](https://discuss.elastic.co/t/doc-id-generation-before-communicating-with-es/5736/2 "2011-11-01T13:05:37Z")
its fine , if ES can give me a set of ID (say 1000 at a time) and guarantee that these ID's never get auto Generated. Thanks Vineeth On Tue, Nov 1, 2011 at 6:27 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote: > Hi , > > I have a system which picks feed , does XYZ data transformations and then > drops to ES. > For each feed there is a feed ID associated. And when the feed reaches ES > , we will receive a related docID. > The feed ID and the doc ID are mapped somewhere to trace the path feed was > picked , transformed and pushed to storage. > > I was wondering if there is some mechanism where a doc ID is generated at > a point long before it reaches ES and i just have to maintain a single ID > for the entire feed. > > So basically i should be able to generate a doc ID much before i push > the feed to elastic Search and i should be able to guarantee that when the > feed reaches the ES , that ID is free (as in no document is present with > that ID). > > Is this possible with ES ? > > Thanks > Vineeth ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [November 1, 2011, 6:12pm UTC](https://discuss.elastic.co/t/doc-id-generation-before-communicating-with-es/5736/3 "2011-11-01T18:12:53Z")
I am not sure I follow what you are trying to do... . On Tue, Nov 1, 2011 at 3:05 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote: > its fine , if ES can give me a set of ID (say 1000 at a time) and > guarantee that these ID's never get auto Generated. > > Thanks > Vineeth > > On Tue, Nov 1, 2011 at 6:27 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote: > > > Hi , > > > > I have a system which picks feed , does XYZ data transformations and > > then drops to ES. > > For each feed there is a feed ID associated. And when the feed reaches ES > > , we will receive a related docID. > > The feed ID and the doc ID are mapped somewhere to trace the path feed > > was picked , transformed and pushed to storage. > > > > I was wondering if there is some mechanism where a doc ID is generated at > > a point long before it reaches ES and i just have to maintain a single ID > > for the entire feed. > > > > So basically i should be able to generate a doc ID much before i push > > the feed to elastic Search and i should be able to guarantee that when the > > feed reaches the ES , that ID is free (as in no document is present with > > that ID). > > > > Is this possible with ES ? > > > > Thanks > > Vineeth ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [November 2, 2011, 8:54am UTC](https://discuss.elastic.co/t/doc-id-generation-before-communicating-with-es/5736/4 "2011-11-02T08:54:12Z")
On Tue, 2011-11-01 at 18:35 +0530, Vineeth Mohan wrote: > its fine , if ES can give me a set of ID (say 1000 at a time) and > guarantee that these ID's never get auto Generated. Vineeth - I released a Perl module to do just this: [https://metacpan.org/module/ElasticSearchX::Sequence](https://metacpan.org/module/ElasticSearchX::Sequence) I explain how it works here: [http://blogs.perl.org/users/clinton\_gormley/2011/10/elasticsearchsequence---a-blazing-fast-ticket-server.html](http://blogs.perl.org/users/clinton_gormley/2011/10/elasticsearchsequence---a-blazing-fast-ticket-server.html) It's simple to do, so you could translate the above into whatever language you are using clint > Thanks > Vineeth > > On Tue, Nov 1, 2011 at 6:27 PM, Vineeth Mohan > [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com) wrote: > Hi , > > ``` > I have a system which picks feed , does XYZ data > transformations and then drops to ES. > For each feed there is a feed ID associated. And when the feed > reaches ES , we will receive a related docID. > The feed ID and the doc ID are mapped somewhere to trace the > path feed was picked , transformed and pushed to storage. > > I was wondering if there is some mechanism where a doc ID is > generated at a point long before it reaches ES and i just have > to maintain a single ID for the entire feed. > > So basically i should be able to generate a doc ID much > before i push the feed to elastic Search and i should be able > to guarantee that when the feed reaches the ES , that ID is > free (as in no document is present with that ID). > > Is this possible with ES ? > > Thanks > Vineeth > > ``` ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [November 3, 2011, 2:31am UTC](https://discuss.elastic.co/t/doc-id-generation-before-communicating-with-es/5736/5 "2011-11-03T02:31:56Z")
Thanks Clinton , that was exactly what i was looking for. Thanks Vineeth On Wed, Nov 2, 2011 at 2:24 PM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com)wrote: > On Tue, 2011-11-01 at 18:35 +0530, Vineeth Mohan wrote: > > > its fine , if ES can give me a set of ID (say 1000 at a time) and > > guarantee that these ID's never get auto Generated. > > Vineeth - I released a Perl module to do just this: > > [https://metacpan.org/module/ElasticSearchX::Sequence](https://metacpan.org/module/ElasticSearchX::Sequence) > > I explain how it works here: > > [ElasticSearch::Sequence - a blazing fast ticket server | Clinton Gormley [blogs.perl.org]](http://blogs.perl.org/users/clinton_gormley/2011/10/elasticsearchsequence---a-blazing-fast-ticket-server.html) > > It's simple to do, so you could translate the above into whatever > language you are using > > clint > > > Thanks > > Vineeth > > > > On Tue, Nov 1, 2011 at 6:27 PM, Vineeth Mohan > > [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com) wrote: > > Hi , > > > > ``` > > I have a system which picks feed , does XYZ data > > transformations and then drops to ES. > > For each feed there is a feed ID associated. And when the feed > > reaches ES , we will receive a related docID. > > The feed ID and the doc ID are mapped somewhere to trace the > > path feed was picked , transformed and pushed to storage. > > > > I was wondering if there is some mechanism where a doc ID is > > generated at a point long before it reaches ES and i just have > > to maintain a single ID for the entire feed. > > > > So basically i should be able to generate a doc ID much > > before i push the feed to elastic Search and i should be able > > to guarantee that when the feed reaches the ES , that ID is > > free (as in no document is present with that ID). > > > > Is this possible with ES ? > > > > Thanks > > Vineeth > > > > ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:50am UTC](https://discuss.elastic.co/t/doc-id-generation-before-communicating-with-es/5736/6 "2017-07-06T03:50:00Z")