# Setting id of document with elasticsearch-hadoop that is not in source document **URL:** \ **Category:** Elasticsearch\ **Created:** [July 7, 2014, 7:48pm UTC](https://discuss.elastic.co/t/setting-id-of-document-with-elasticsearch-hadoop-that-is-not-in-source-document/18509 "2014-07-07T19:48:06Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Brian\_Thomas\_2](https://avatars.discourse-cdn.com/v4/letter/b/e56c9b/32.png) [@Brian\_Thomas\_2](https://discuss.elastic.co/u/Brian_Thomas_2)\ **Post date:** [July 7, 2014, 7:48pm UTC](https://discuss.elastic.co/t/setting-id-of-document-with-elasticsearch-hadoop-that-is-not-in-source-document/18509/1 "2014-07-07T19:48:06Z")
I am trying to update an elasticsearch index using elasticsearch-hadoop. I am aware of the _es.mapping.id_ configuration where you can specify that field in the document to use as an id, but in my case the source document does not have the id (I used elasticsearch's autogenerated id when indexing the document). Is it possible to specify the id to update without having the add a new field to the MapWritable object? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ce6161ad-d442-4ffb-9162-114cb8cd76dd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ce6161ad-d442-4ffb-9162-114cb8cd76dd%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [July 10, 2014, 9:48pm UTC](https://discuss.elastic.co/t/setting-id-of-document-with-elasticsearch-hadoop-that-is-not-in-source-document/18509/2 "2014-07-10T21:48:53Z")
You need to specify the id of the document you want to update somehow. Since in es-hadoop things are batch focused, each doc needs its own id specified somehow hence the use of 'es.mapping.id' to indicate its value. Is there a reason why this approach does not work for you - any alternatives that you thought of? Cheers, On 7/7/14 10:48 PM, Brian Thomas wrote: > I am trying to update an elasticsearch index using elasticsearch-hadoop. I am aware of the _es.mapping.id_ > configuration where you can specify that field in the document to use as an id, but in my case the source document does > not have the id (I used elasticsearch's autogenerated id when indexing the document). Is it possible to specify the id > to update without having the add a new field to the MapWritable object? > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/ce6161ad-d442-4ffb-9162-114cb8cd76dd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ce6161ad-d442-4ffb-9162-114cb8cd76dd%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/ce6161ad-d442-4ffb-9162-114cb8cd76dd%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ce6161ad-d442-4ffb-9162-114cb8cd76dd%40googlegroups.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/53BF0A45.7000403%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/53BF0A45.7000403%40gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Brian\_Thomas\_2](https://avatars.discourse-cdn.com/v4/letter/b/e56c9b/32.png) [@Brian\_Thomas\_2](https://discuss.elastic.co/u/Brian_Thomas_2)\ **Post date:** [July 11, 2014, 6:31pm UTC](https://discuss.elastic.co/t/setting-id-of-document-with-elasticsearch-hadoop-that-is-not-in-source-document/18509/3 "2014-07-11T18:31:47Z")
I was just curious if there was a way of doing this without doing this, I can add the field if necessary. For alternatives, what if in addition to es.mapping.id, there is another property available also, like es.mapping.id.include.in.src where you could specify whether the src field actually gets included in the source document. In elasticsearch, you can create and update documents without having to include the id in the source document, so I think it would make sense to be able to do that with elasticsearch-hadoop also. On Thursday, July 10, 2014 5:49:18 PM UTC-4, Costin Leau wrote: > You need to specify the id of the document you want to update somehow. > Since in es-hadoop things are batch focused, each > doc needs its own id specified somehow hence the use of 'es.mapping.id' > to indicate its value. > Is there a reason why this approach does not work for you - any > alternatives that you thought of? > > Cheers, > > On 7/7/14 10:48 PM, Brian Thomas wrote: > > > I am trying to update an elasticsearch index using elasticsearch-hadoop. > > I am aware of the _es.mapping.id_ > > configuration where you can specify that field in the document to use as > > an id, but in my case the source document does > > not have the id (I used elasticsearch's autogenerated id when indexing > > the document). Is it possible to specify the id > > to update without having the add a new field to the MapWritable object? > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \ \ > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \\>. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/ce6161ad-d442-4ffb-9162-114cb8cd76dd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ce6161ad-d442-4ffb-9162-114cb8cd76dd%40googlegroups.com) > > > \< > > [https://groups.google.com/d/msgid/elasticsearch/ce6161ad-d442-4ffb-9162-114cb8cd76dd%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ce6161ad-d442-4ffb-9162-114cb8cd76dd%40googlegroups.com?utm_medium=email&utm_source=footer)\>. > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/77259ed3-a896-47cc-9304-cc32046756ad%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/77259ed3-a896-47cc-9304-cc32046756ad%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Brian\_Thomas\_2](https://avatars.discourse-cdn.com/v4/letter/b/e56c9b/32.png) [@Brian\_Thomas\_2](https://discuss.elastic.co/u/Brian_Thomas_2)\ **Post date:** [July 11, 2014, 6:36pm UTC](https://discuss.elastic.co/t/setting-id-of-document-with-elasticsearch-hadoop-that-is-not-in-source-document/18509/4 "2014-07-11T18:36:09Z")
I was just curious if there was a way of doing this without doing this, I can add the field if necessary. For alternatives, what if in addition to es.mapping.id, there is another property available also, like es.mapping.id.exlude that will not include the id field in the source document. In elasticsearch, you can create and update documents without having to include the id in the source document, so I think it would make sense to be able to do that with elasticsearch-hadoop also. On Thursday, July 10, 2014 5:49:18 PM UTC-4, Costin Leau wrote: > You need to specify the id of the document you want to update somehow. > Since in es-hadoop things are batch focused, each > doc needs its own id specified somehow hence the use of 'es.mapping.id' > to indicate its value. > Is there a reason why this approach does not work for you - any > alternatives that you thought of? > > Cheers, > > On 7/7/14 10:48 PM, Brian Thomas wrote: > > > I am trying to update an elasticsearch index using elasticsearch-hadoop. > > I am aware of the _es.mapping.id_ > > configuration where you can specify that field in the document to use as > > an id, but in my case the source document does > > not have the id (I used elasticsearch's autogenerated id when indexing > > the document). Is it possible to specify the id > > to update without having the add a new field to the MapWritable object? > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \ \ > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \\>. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/ce6161ad-d442-4ffb-9162-114cb8cd76dd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ce6161ad-d442-4ffb-9162-114cb8cd76dd%40googlegroups.com) > > > \< > > [https://groups.google.com/d/msgid/elasticsearch/ce6161ad-d442-4ffb-9162-114cb8cd76dd%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ce6161ad-d442-4ffb-9162-114cb8cd76dd%40googlegroups.com?utm_medium=email&utm_source=footer)\>. > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2c6753aa-c459-489b-9f86-6803a5616718%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/2c6753aa-c459-489b-9f86-6803a5616718%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [July 11, 2014, 8:39pm UTC](https://discuss.elastic.co/t/setting-id-of-document-with-elasticsearch-hadoop-that-is-not-in-source-document/18509/5 "2014-07-11T20:39:23Z")
Hi, I've opened up issue #230 to address your use case. Rather than offering a dedicated field for the ID, I opted to introduce an "include", "exclude" option to select (or remove) certain fields from a document before being saved to es. This will basically allow documents to be filtered and thus exclude the 'metadata' or fields that are not needed in ES directly through es-hadoop. Cheers, On Fri, Jul 11, 2014 at 9:36 PM, Brian Thomas [brianjthomas85@gmail.com](mailto:brianjthomas85@gmail.com) wrote: > I was just curious if there was a way of doing this without doing this, I > can add the field if necessary. > > For alternatives, what if in addition to es.mapping.id, there is another > property available also, like es.mapping.id.exlude that will not include > the id field in the source document. In elasticsearch, you can create and > update documents without having to include the id in the source document, > so I think it would make sense to be able to do that with > elasticsearch-hadoop also. > > On Thursday, July 10, 2014 5:49:18 PM UTC-4, Costin Leau wrote: > > > You need to specify the id of the document you want to update somehow. > > Since in es-hadoop things are batch focused, each > > doc needs its own id specified somehow hence the use of 'es.mapping.id' > > to indicate its value. > > Is there a reason why this approach does not work for you - any > > alternatives that you thought of? > > > > Cheers, > > > > On 7/7/14 10:48 PM, Brian Thomas wrote: > > > > > I am trying to update an elasticsearch index using > > > elasticsearch-hadoop. I am aware of the _es.mapping.id_ > > > configuration where you can specify that field in the document to use > > > as an id, but in my case the source document does > > > not have the id (I used elasticsearch's autogenerated id when indexing > > > the document). Is it possible to specify the id > > > to update without having the add a new field to the MapWritable object? > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \ > > [unsubscribe@googlegroups.com](mailto:unsubscribe@googlegroups.com)\>. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/ce6161ad-](https://groups.google.com/d/msgid/elasticsearch/ce6161ad-) > > > d442-4ffb-9162-114cb8cd76dd%[40googlegroups.com](http://40googlegroups.com) > > > \<[https://groups.google.com/d/msgid/elasticsearch/ce6161ad-](https://groups.google.com/d/msgid/elasticsearch/ce6161ad-) > > > d442-4ffb-9162-114cb8cd76dd%[40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players](http://40googlegroups.com?utm_medium=) > > > email&utm\_source=footer\>. > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > Costin > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/2c6753aa-c459-489b-9f86-6803a5616718%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/2c6753aa-c459-489b-9f86-6803a5616718%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/2c6753aa-c459-489b-9f86-6803a5616718%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2c6753aa-c459-489b-9f86-6803a5616718%40googlegroups.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJogdmd-EBAvd7hC3CZs%2BhjoohNuC\_%2B%3Da%2B2k\_kqKeKO9-jLFmA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJogdmd-EBAvd7hC3CZs%2BhjoohNuC_%2B%3Da%2B2k_kqKeKO9-jLFmA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![James\_Campbell](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_campbell/32/1411_2.png) [@James\_Campbell](https://discuss.elastic.co/u/James_Campbell)\ **Post date:** [July 13, 2014, 12:27pm UTC](https://discuss.elastic.co/t/setting-id-of-document-with-elasticsearch-hadoop-that-is-not-in-source-document/18509/6 "2014-07-13T12:27:46Z")
Thanks for suggesting this option, I would also definitely like to have an exclude option for fields that I currently have to include only to set the \_id, \_type, and index, resulting in unnecessary fields in \_source. On Fri, Jul 11, 2014 at 4:39 PM, Costin Leau [costin.leau@gmail.com](mailto:costin.leau@gmail.com) wrote: > Hi, > > I've opened up issue #230 to address your use case. Rather than offering a > dedicated field for the ID, I opted to introduce an "include", "exclude" > option to select (or remove) certain fields from a document before being > saved to es. This will basically allow documents to be filtered and thus > exclude the 'metadata' or fields that are not needed in ES directly through > es-hadoop. > > Cheers, > > On Fri, Jul 11, 2014 at 9:36 PM, Brian Thomas [brianjthomas85@gmail.com](mailto:brianjthomas85@gmail.com) > wrote: > > > I was just curious if there was a way of doing this without doing this, I > > can add the field if necessary. > > > > For alternatives, what if in addition to es.mapping.id, there is another > > property available also, like es.mapping.id.exlude that will not include > > the id field in the source document. In elasticsearch, you can create and > > update documents without having to include the id in the source document, > > so I think it would make sense to be able to do that with > > elasticsearch-hadoop also. > > > > On Thursday, July 10, 2014 5:49:18 PM UTC-4, Costin Leau wrote: > > > > > You need to specify the id of the document you want to update somehow. > > > Since in es-hadoop things are batch focused, each > > > doc needs its own id specified somehow hence the use of 'es.mapping.id' > > > to indicate its value. > > > Is there a reason why this approach does not work for you - any > > > alternatives that you thought of? > > > > > > Cheers, > > > > > > On 7/7/14 10:48 PM, Brian Thomas wrote: > > > > > > > I am trying to update an elasticsearch index using > > > > elasticsearch-hadoop. I am aware of the _es.mapping.id_ > > > > configuration where you can specify that field in the document to use > > > > as an id, but in my case the source document does > > > > not have the id (I used elasticsearch's autogenerated id when indexing > > > > the document). Is it possible to specify the id > > > > to update without having the add a new field to the MapWritable > > > > object? > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \ > > > [unsubscribe@googlegroups.com](mailto:unsubscribe@googlegroups.com)\>. > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/ce6161ad-](https://groups.google.com/d/msgid/elasticsearch/ce6161ad-) > > > > d442-4ffb-9162-114cb8cd76dd%[40googlegroups.com](http://40googlegroups.com) > > > > \<[https://groups.google.com/d/msgid/elasticsearch/ce6161ad-](https://groups.google.com/d/msgid/elasticsearch/ce6161ad-) > > > > d442-4ffb-9162-114cb8cd76dd%[40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players](http://40googlegroups.com?utm_medium=) > > > > email&utm\_source=footer\>. > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > Costin > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/2c6753aa-c459-489b-9f86-6803a5616718%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/2c6753aa-c459-489b-9f86-6803a5616718%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/2c6753aa-c459-489b-9f86-6803a5616718%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2c6753aa-c459-489b-9f86-6803a5616718%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/zynzkAIWzp0/unsubscribe](https://groups.google.com/d/topic/elasticsearch/zynzkAIWzp0/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAJogdmd-EBAvd7hC3CZs%2BhjoohNuC\_%2B%3Da%2B2k\_kqKeKO9-jLFmA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJogdmd-EBAvd7hC3CZs%2BhjoohNuC_%2B%3Da%2B2k_kqKeKO9-jLFmA%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAJogdmd-EBAvd7hC3CZs%2BhjoohNuC\_%2B%3Da%2B2k\_kqKeKO9-jLFmA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAJogdmd-EBAvd7hC3CZs%2BhjoohNuC_%2B%3Da%2B2k_kqKeKO9-jLFmA%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CA%2BAQu3yWpd36vnCxHbUi83GEGue2WjYQ%2B%2Bj\_7xRWrsnSEeCvBg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CA%2BAQu3yWpd36vnCxHbUi83GEGue2WjYQ%2B%2Bj_7xRWrsnSEeCvBg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:16am UTC](https://discuss.elastic.co/t/setting-id-of-document-with-elasticsearch-hadoop-that-is-not-in-source-document/18509/7 "2017-07-06T01:16:13Z")