# \[Hadoop\]\[pig\] How to set the document id? **URL:** \ **Category:** Elasticsearch\ **Created:** [December 1, 2013, 3:21pm UTC](https://discuss.elastic.co/t/hadoop-pig-how-to-set-the-document-id/14653 "2013-12-01T15:21:01Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Niels\_Basjes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niels_basjes/32/109091_2.png) [@Niels\_Basjes](https://discuss.elastic.co/u/Niels_Basjes)\ **Post date:** [December 1, 2013, 3:21pm UTC](https://discuss.elastic.co/t/hadoop-pig-how-to-set-the-document-id/14653/1 "2013-12-01T15:21:01Z")
I've been doing some PoC applications using the ES STORE functionality that allows me to load data from pub directly into ES. In some cases after running an job I want to redo the same data but with some small changes to the job. In those scenarios I want to simply update the existing documents with a newer version (having a few changed fields). I think there way to do that is by manually setting the document id. So far I've not been able to figure out how to set the document id for my pig script. A simple working example would really help me out. Thanks. Niels Basjes -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3766096f-d5b9-4b79-8adf-cb1f44dfe2b1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3766096f-d5b9-4b79-8adf-cb1f44dfe2b1%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [December 1, 2013, 9:01pm UTC](https://discuss.elastic.co/t/hadoop-pig-how-to-set-the-document-id/14653/2 "2013-12-01T21:01:13Z")
This functionality is available in master. Define 'es.mapping.id' property which can point to either a field name or an absolute value (specified in \< > ). > From the tests > B = FOREACH A GENERATE id, name, TOBAG(url, picture) AS links > STORE B INTO 'pig/update' USING org.elasticsearch.hadoop.pig.ESStorage(' > es.mapping.id=id') Note that you can specify also the write operation - whether it's index (default), create or update (doc upsert). For your case, the default should work just fine. Cheers, On Sun, Dec 1, 2013 at 5:21 PM, Niels Basjes [niels@basj.es](mailto:niels@basj.es) wrote: > I've been doing some PoC applications using the ES STORE functionality > that allows me to load data from pub directly into ES. > In some cases after running an job I want to redo the same data but with > some small changes to the job. In those scenarios I want to simply update > the existing documents with a newer version (having a few changed fields). > > I think there way to do that is by manually setting the document id. So > far I've not been able to figure out how to set the document id for my pig > script. > > A simple working example would really help me out. > > Thanks. > > Niels Basjes > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/3766096f-d5b9-4b79-8adf-cb1f44dfe2b1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3766096f-d5b9-4b79-8adf-cb1f44dfe2b1%40googlegroups.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJogdmdpgpYwo7pNcLXrCebotatSW3KE3L1G-Yr\_voadiHq8rg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJogdmdpgpYwo7pNcLXrCebotatSW3KE3L1G-Yr_voadiHq8rg%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Niels\_Basjes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niels_basjes/32/109091_2.png) [@Niels\_Basjes](https://discuss.elastic.co/u/Niels_Basjes)\ **Post date:** [December 2, 2013, 7:13pm UTC](https://discuss.elastic.co/t/hadoop-pig-how-to-set-the-document-id/14653/3 "2013-12-02T19:13:44Z")
Thanks I'll try to build from master a version of the plugin later this week. Niels -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/23def31c-d00d-40b6-883b-66ac78db13f6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/23def31c-d00d-40b6-883b-66ac78db13f6%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [December 2, 2013, 7:42pm UTC](https://discuss.elastic.co/t/hadoop-pig-how-to-set-the-document-id/14653/4 "2013-12-02T19:42:28Z")
The snapshots are published daily to a maven repo so you can just point your maven tool to it and are ready to go [1] [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/install.html) On 02/12/2013 9:13 PM, Niels Basjes wrote: > Thanks I'll try to build from master a version of the plugin later this week. > > Niels -- Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/529CE2A4.1080005%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/529CE2A4.1080005%40gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Niels\_Basjes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niels_basjes/32/109091_2.png) [@Niels\_Basjes](https://discuss.elastic.co/u/Niels_Basjes)\ **Post date:** [December 2, 2013, 7:53pm UTC](https://discuss.elastic.co/t/hadoop-pig-how-to-set-the-document-id/14653/5 "2013-12-02T19:53:30Z")
Hi, Op maandag 2 december 2013 20:42:28 UTC+1 schreef Costin Leau: > The snapshots are published daily to a maven repo so you can just point > your maven tool to it and are ready to go [1] > > [1] > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/install.html) Thanks. From pig I simply need the jar with all the stuff, so Maven doesn't do that for me (not easily). I did some browsing and found this which is just as good for me: [http://oss.sonatype.org/content/repositories/snapshots/org/elasticsearch/elasticsearch-hadoop/1.3.0.BUILD-SNAPSHOT/](http://oss.sonatype.org/content/repositories/snapshots/org/elasticsearch/elasticsearch-hadoop/1.3.0.BUILD-SNAPSHOT/) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5ef594fd-f0a0-4b85-ab17-578ea26b6999%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5ef594fd-f0a0-4b85-ab17-578ea26b6999%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [December 2, 2013, 8:08pm UTC](https://discuss.elastic.co/t/hadoop-pig-how-to-set-the-document-id/14653/6 "2013-12-02T20:08:41Z")
Not sure what you mean by "the jar with all the stuff" - I was suggesting using a maven-compatible tool to download the jar easily rather then manually. You can then manipulate the jar any way you want. Cheers, On 02/12/2013 9:53 PM, Niels Basjes wrote: > Hi, > > Op maandag 2 december 2013 20:42:28 UTC+1 schreef Costin Leau: > > ``` > The snapshots are published daily to a maven repo so you can just point your maven tool to it and are ready to go [1] > > [1] http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/install.html > > > ``` > > Thanks. > From pig I simply need the jar with all the stuff, so Maven doesn't do that for me (not easily). > I did some browsing and found this which is just as good for me: > [Index of /repositories/snapshots/org/elasticsearch/elasticsearch-hadoop/1.3.0.BUILD-SNAPSHOT](http://oss.sonatype.org/content/repositories/snapshots/org/elasticsearch/elasticsearch-hadoop/1.3.0.BUILD-SNAPSHOT/) > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/5ef594fd-f0a0-4b85-ab17-578ea26b6999%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5ef594fd-f0a0-4b85-ab17-578ea26b6999%40googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/529CE8C9.105%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/529CE8C9.105%40gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:03am UTC](https://discuss.elastic.co/t/hadoop-pig-how-to-set-the-document-id/14653/7 "2017-07-06T02:03:47Z")