# \[hadoop\] Pipelining Hadoop/Spark with ElasticSearch **URL:** \ **Category:** Elasticsearch\ **Created:** [October 24, 2013, 4:53pm UTC](https://discuss.elastic.co/t/hadoop-pipelining-hadoop-spark-with-elasticsearch/14093 "2013-10-24T16:53:55Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Han\_JU](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/han_ju/32/962_2.png) [@Han\_JU](https://discuss.elastic.co/u/Han_JU)\ **Post date:** [October 24, 2013, 4:53pm UTC](https://discuss.elastic.co/t/hadoop-pipelining-hadoop-spark-with-elasticsearch/14093/1 "2013-10-24T16:53:55Z")
Hi, I'm trying to write hadoop aggregation results to ES. Say I've K, V for key and value classes respectively. According to elasticsearch-hadoop api/blog, the key is ignored and the value should be a Map\. I'm a little bit confused here: do I need an extra map job to convert my (K, V) to (Null, Map\) ? Is there any complete examples of using hadoop and ES together? Thanks. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![mattweber](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattweber/32/44940_2.png) [@mattweber](https://discuss.elastic.co/u/mattweber)\ **Post date:** [October 24, 2013, 5:04pm UTC](https://discuss.elastic.co/t/hadoop-pipelining-hadoop-spark-with-elasticsearch/14093/2 "2013-10-24T17:04:43Z")
Have you tried the elasticsearch-hadoop plugin? There is good documentation on the website. > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. > **[GitHub - elastic/elasticsearch-hadoop: Elasticsearch real-time search and...](https://github.com/elastic/elasticsearch-hadoop)** > > :elephant: Elasticsearch real-time search and analytics natively integrated with Hadoop - GitHub - elastic/elasticsearch-hadoop: Elasticsearch real-time search and analytics natively integrated wi... Thanks, Matt Weber On Thu, Oct 24, 2013 at 9:53 AM, Han JU [ju.han.felix@gmail.com](mailto:ju.han.felix@gmail.com) wrote: > Hi, > > I'm trying to write hadoop aggregation results to ES. > Say I've K, V for key and value classes respectively. According to > elasticsearch-hadoop api/blog, the key is ignored and the value should be a > Map\. > I'm a little bit confused here: do I need an extra map job to convert my > (K, V) to (Null, Map\) ? > Is there any complete examples of using hadoop and ES together? > > Thanks. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [October 24, 2013, 5:05pm UTC](https://discuss.elastic.co/t/hadoop-pipelining-hadoop-spark-with-elasticsearch/14093/3 "2013-10-24T17:05:31Z")
Hi, I replied on IRC but you left. See the docs here [1]. The value represents your document and since it might contain multiple fields, ESOuputFormat expects a Map (MapWritable) which contains the actual document. Say your doc is something like { foo: 123 } then your map would be [Text("foo"):new LongWritable(123)]. The docs provides more information about the Writable types supported (basically all of them) and their equivalent ES types. [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/mapreduce.html) On 24/10/2013 7:53 PM, Han JU wrote: > Hi, > > I'm trying to write hadoop aggregation results to ES. > Say I've K, V for key and value classes respectively. According to elasticsearch-hadoop api/blog, the key is ignored and > the value should be a Map\. > I'm a little bit confused here: do I need an extra map job to convert my (K, V) to (Null, Map\) ? > Is there any complete examples of using hadoop and ES together? > > Thanks. > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Han\_JU](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/han_ju/32/962_2.png) [@Han\_JU](https://discuss.elastic.co/u/Han_JU)\ **Post date:** [October 25, 2013, 1:26pm UTC](https://discuss.elastic.co/t/hadoop-pipelining-hadoop-spark-with-elasticsearch/14093/4 "2013-10-25T13:26:16Z")
Thanks. Seems like I misunderstand something. Now I managed to push documents to ES, and I'd like to know if these are supported by current version of elasticsearch-binding: - specifying id for index. Now the "\_id" for the documents pushed are auto generated - the update api Thanks. 在 2013年10月24日星期四UTC+2下午7时05分31秒，Costin Leau写道： > Hi, > > I replied on IRC but you left. See the docs here [1]. The value represents > your document and since it might contain > multiple fields, ESOuputFormat expects a Map (MapWritable) which contains > the actual document. Say your doc is something > like { foo: 123 } then your map would be [Text("foo"):new > LongWritable(123)]. > > The docs provides more information about the Writable types supported > (basically all of them) and their equivalent ES types. > > [1] > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/mapreduce.html) > > On 24/10/2013 7:53 PM, Han JU wrote: > > > Hi, > > > > I'm trying to write hadoop aggregation results to ES. > > Say I've K, V for key and value classes respectively. According to > > elasticsearch-hadoop api/blog, the key is ignored and > > the value should be a Map\. > > I'm a little bit confused here: do I need an extra map job to convert my > > (K, V) to (Null, Map\) ? > > Is there any complete examples of using hadoop and ES together? > > > > Thanks. > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [October 25, 2013, 2:10pm UTC](https://discuss.elastic.co/t/hadoop-pipelining-hadoop-spark-with-elasticsearch/14093/5 "2013-10-25T14:10:05Z")
On 25/10/2013 4:26 PM, Han JU wrote: > Thanks. Seems like I misunderstand something. > > Now I managed to push documents to ES, and I'd like to know if these are supported by current version of > elasticsearch-binding: I assume you mean elasticsearch-hadoop. > - specifying id for index. Now the "\_id" for the documents pushed are auto generated > - the update api This is being currently worked on and we should have something in trunk by next week. > Thanks. > > 在 2013年10月24日星期四UTC+2下午7时05分31秒，Costin Leau写道： > > ``` > Hi, > > I replied on IRC but you left. See the docs here [1]. The value represents your document and since it might contain > multiple fields, ESOuputFormat expects a Map (MapWritable) which contains the actual document. Say your doc is > something > like { foo: 123 } then your map would be [Text("foo"):new LongWritable(123)]. > > The docs provides more information about the Writable types supported (basically all of them) and their equivalent > ES types. > > [1] http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/mapreduce.html > > > On 24/10/2013 7:53 PM, Han JU wrote: > > Hi, > > > > I'm trying to write hadoop aggregation results to ES. > > Say I've K, V for key and value classes respectively. According to elasticsearch-hadoop api/blog, the key is ignored and > > the value should be a Map. > > I'm a little bit confused here: do I need an extra map job to convert my (K, V) to (Null, Map) ? > > Is there any complete examples of using hadoop and ES together? > > > > Thanks. > > > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to > >elasticsearc...@googlegroups.com . > > For more options, visithttps://groups.google.com/groups/opt_out . > > -- > Costin > > ``` > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![aarthi1890](https://avatars.discourse-cdn.com/v4/letter/a/f05b48/32.png) [@aarthi1890](https://discuss.elastic.co/u/aarthi1890)\ **Post date:** [September 10, 2014, 6:01am UTC](https://discuss.elastic.co/t/hadoop-pipelining-hadoop-spark-with-elasticsearch/14093/6 "2014-09-10T06:01:02Z")
Hi Wanted to know if the auto generated id has been committed. Thanks Aarthi On Friday, 25 October 2013 19:40:05 UTC+5:30, Costin Leau wrote: > On 25/10/2013 4:26 PM, Han JU wrote: > > > Thanks. Seems like I misunderstand something. > > > > Now I managed to push documents to ES, and I'd like to know if these are > > supported by current version of > > elasticsearch-binding: > > I assume you mean elasticsearch-hadoop. > > > - specifying id for index. Now the "\_id" for the documents pushed are > > auto generated > > - the update api > > This is being currently worked on and we should have something in trunk by > next week. > > > Thanks. > > > > 在 2013年10月24日星期四UTC+2下午7时05分31秒，Costin Leau写道： > > > > ``` > > Hi, > > > > I replied on IRC but you left. See the docs here [1]. The value > > > > ``` > > represents your document and since it might contain > > > ``` > > multiple fields, ESOuputFormat expects a Map (MapWritable) which > > > > ``` > > contains the actual document. Say your doc is > > > ``` > > something > > like { foo: 123 } then your map would be [Text("foo"):new > > > > ``` > > LongWritable(123)]. > > > ``` > > The docs provides more information about the Writable types > > > > ``` > > supported (basically all of them) and their equivalent > > > ``` > > ES types. > > > > [1] > > > > ``` > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/mapreduce.html) > > > ``` > > < > > > > ``` > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/mapreduce.html)\> > > > ``` > > On 24/10/2013 7:53 PM, Han JU wrote: > > > Hi, > > > > > > I'm trying to write hadoop aggregation results to ES. > > > Say I've K, V for key and value classes respectively. According to > > > > ``` > > elasticsearch-hadoop api/blog, the key is ignored and > > > ``` > > > the value should be a Map. > > > I'm a little bit confused here: do I need an extra map job to > > > > ``` > > convert my (K, V) to (Null, Map\) ? > > > ``` > > > Is there any complete examples of using hadoop and ES together? > > > > > > Thanks. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > > ``` > > Groups "elasticsearch" group. > > > ``` > > > To unsubscribe from this group and stop receiving emails from it, > > > > ``` > > send an email to > > > ``` > > >elasticsearc...@googlegroups.com . > > > For more options, visithttps://groups.google.com/groups/opt_out < > > > > ``` > > [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)\>. > > > ``` > > -- > > Costin > > > > ``` > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ed879b28-a983-4fb7-bf40-bf6fc0eff68d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ed879b28-a983-4fb7-bf40-bf6fc0eff68d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![aarthi1890](https://avatars.discourse-cdn.com/v4/letter/a/f05b48/32.png) [@aarthi1890](https://discuss.elastic.co/u/aarthi1890)\ **Post date:** [September 10, 2014, 6:02am UTC](https://discuss.elastic.co/t/hadoop-pipelining-hadoop-spark-with-elasticsearch/14093/7 "2014-09-10T06:02:59Z")
Hi Just wanted to know if the code for changing auto generated id has been committed or is it yet to be changed? I am using elasticsearch-spark\_2.10.Beta 1 version. Thanks Aarthi Thanks On Friday, 25 October 2013 19:40:05 UTC+5:30, Costin Leau wrote: > On 25/10/2013 4:26 PM, Han JU wrote: > > > Thanks. Seems like I misunderstand something. > > > > Now I managed to push documents to ES, and I'd like to know if these are > > supported by current version of > > elasticsearch-binding: > > I assume you mean elasticsearch-hadoop. > > > - specifying id for index. Now the "\_id" for the documents pushed are > > auto generated > > - the update api > > This is being currently worked on and we should have something in trunk by > next week. > > > Thanks. > > > > 在 2013年10月24日星期四UTC+2下午7时05分31秒，Costin Leau写道： > > > > ``` > > Hi, > > > > I replied on IRC but you left. See the docs here [1]. The value > > > > ``` > > represents your document and since it might contain > > > ``` > > multiple fields, ESOuputFormat expects a Map (MapWritable) which > > > > ``` > > contains the actual document. Say your doc is > > > ``` > > something > > like { foo: 123 } then your map would be [Text("foo"):new > > > > ``` > > LongWritable(123)]. > > > ``` > > The docs provides more information about the Writable types > > > > ``` > > supported (basically all of them) and their equivalent > > > ``` > > ES types. > > > > [1] > > > > ``` > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/mapreduce.html) > > > ``` > > < > > > > ``` > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/mapreduce.html)\> > > > ``` > > On 24/10/2013 7:53 PM, Han JU wrote: > > > Hi, > > > > > > I'm trying to write hadoop aggregation results to ES. > > > Say I've K, V for key and value classes respectively. According to > > > > ``` > > elasticsearch-hadoop api/blog, the key is ignored and > > > ``` > > > the value should be a Map. > > > I'm a little bit confused here: do I need an extra map job to > > > > ``` > > convert my (K, V) to (Null, Map\) ? > > > ``` > > > Is there any complete examples of using hadoop and ES together? > > > > > > Thanks. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > > ``` > > Groups "elasticsearch" group. > > > ``` > > > To unsubscribe from this group and stop receiving emails from it, > > > > ``` > > send an email to > > > ``` > > >elasticsearc...@googlegroups.com . > > > For more options, visithttps://groups.google.com/groups/opt_out < > > > > ``` > > [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)\>. > > > ``` > > -- > > Costin > > > > ``` > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3a82d348-066c-421e-996e-69b81455f175%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3a82d348-066c-421e-996e-69b81455f175%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [September 10, 2014, 8:53am UTC](https://discuss.elastic.co/t/hadoop-pipelining-hadoop-spark-with-elasticsearch/14093/8 "2014-09-10T08:53:32Z")
One can specify the id for each document for quite some time now, through `es.mapping.id` parameter [1] - simply point it to the field containing the ID and you're good to go. > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. On 9/10/14 9:02 AM, aarthi ranganathan wrote: > Hi > Just wanted to know if the code for changing auto generated id has been committed or is it yet to be changed? I am using > elasticsearch-spark\_2.10.Beta 1 version. > Thanks > Aarthi > Thanks > On Friday, 25 October 2013 19:40:05 UTC+5:30, Costin Leau wrote: > > ``` > On 25/10/2013 4:26 PM, Han JU wrote: > > Thanks. Seems like I misunderstand something. > > > > Now I managed to push documents to ES, and I'd like to know if these are supported by current version of > > elasticsearch-binding: > > > > I assume you mean elasticsearch-hadoop. > > > - specifying id for index. Now the "_id" for the documents pushed are auto generated > > - the update api > > > > This is being currently worked on and we should have something in trunk by next week. > > > Thanks. > > > > 在 2013年10月24日星期四UTC+2下午7时05分31秒，Costin Leau写道： > > > > Hi, > > > > I replied on IRC but you left. See the docs here [1]. The value represents your document and since it might contain > > multiple fields, ESOuputFormat expects a Map (MapWritable) which contains the actual document. Say your doc is > > something > > like { foo: 123 } then your map would be [Text("foo"):new LongWritable(123)]. > > > > The docs provides more information about the Writable types supported (basically all of them) and their equivalent > > ES types. > > > > [1]http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/mapreduce.html > > > > > > > > On 24/10/2013 7:53 PM, Han JU wrote: > > > Hi, > > > > > > I'm trying to write hadoop aggregation results to ES. > > > Say I've K, V for key and value classes respectively. According to elasticsearch-hadoop api/blog, the key is ignored and > > > the value should be a Map. > > > I'm a little bit confused here: do I need an extra map job to convert my (K, V) to (Null, Map) ? > > > Is there any complete examples of using hadoop and ES together? > > > > > > Thanks. > > > > > > -- > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an email to > > >elasticsearc...@googlegroups.com . > > > For more options, visithttps://groups.google.com/groups/opt_out >. > > > > -- > > Costin > > > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to > >elasticsearc...@googlegroups.com . > > For more options, visithttps://groups.google.com/groups/opt_out . > > -- > Costin > > ``` > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/3a82d348-066c-421e-996e-69b81455f175%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3a82d348-066c-421e-996e-69b81455f175%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/3a82d348-066c-421e-996e-69b81455f175%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/3a82d348-066c-421e-996e-69b81455f175%40googlegroups.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5410118C.1060907%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/5410118C.1060907%40gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![aarthi1890](https://avatars.discourse-cdn.com/v4/letter/a/f05b48/32.png) [@aarthi1890](https://discuss.elastic.co/u/aarthi1890)\ **Post date:** [September 10, 2014, 10:29am UTC](https://discuss.elastic.co/t/hadoop-pipelining-hadoop-spark-with-elasticsearch/14093/9 "2014-09-10T10:29:20Z")
Thanks for the reply Costin. On Wednesday, 10 September 2014 14:23:57 UTC+5:30, Costin Leau wrote: > One can specify the id for each document for quite some time now, through ` es.mapping.id` parameter [1] - simply point > it to the field containing the ID and you're good to go. > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/2.1.Beta/configuration.html#_mapping) > > On 9/10/14 9:02 AM, aarthi ranganathan wrote: > > > Hi > > Just wanted to know if the code for changing auto generated id has been > > committed or is it yet to be changed? I am using > > elasticsearch-spark\_2.10.Beta 1 version. > > Thanks > > Aarthi > > Thanks > > On Friday, 25 October 2013 19:40:05 UTC+5:30, Costin Leau wrote: > > > > ``` > > On 25/10/2013 4:26 PM, Han JU wrote: > > > Thanks. Seems like I misunderstand something. > > > > > > Now I managed to push documents to ES, and I'd like to know if > > > > ``` > > these are supported by current version of > > > ``` > > > elasticsearch-binding: > > > > > > > I assume you mean elasticsearch-hadoop. > > > > > - specifying id for index. Now the "_id" for the documents pushed > > > > ``` > > are auto generated > > > ``` > > > - the update api > > > > > > > This is being currently worked on and we should have something in > > > > ``` > > trunk by next week. > > > ``` > > > Thanks. > > > > > > 在 2013年10月24日星期四UTC+2下午7时05分31秒，Costin Leau写道： > > > > > > Hi, > > > > > > I replied on IRC but you left. See the docs here [1]. The > > > > ``` > > value represents your document and since it might contain > > > ``` > > > multiple fields, ESOuputFormat expects a Map (MapWritable) > > > > ``` > > which contains the actual document. Say your doc is > > > ``` > > > something > > > like { foo: 123 } then your map would be [Text("foo"):new > > > > ``` > > LongWritable(123)]. > > > ``` > > > > > > The docs provides more information about the Writable types > > > > ``` > > supported (basically all of them) and their equivalent > > > ``` > > > ES types. > > > > > > [1] > > > > ``` > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/mapreduce.html) > > > ``` > > < > > > > ``` > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/mapreduce.html)\> > > > ``` > > > < > > > > ``` > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/mapreduce.html) > > > ``` > > < > > > > ``` > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/mapreduce.html)\>\> > > > ``` > > > > > > On 24/10/2013 7:53 PM, Han JU wrote: > > > > Hi, > > > > > > > > I'm trying to write hadoop aggregation results to ES. > > > > Say I've K, V for key and value classes respectively. > > > > ``` > > According to elasticsearch-hadoop api/blog, the key is ignored and > > > ``` > > > > the value should be a Map. > > > > I'm a little bit confused here: do I need an extra map job > > > > ``` > > to convert my (K, V) to (Null, Map\) ? > > > ``` > > > > Is there any complete examples of using hadoop and ES > > > > ``` > > together? > > > ``` > > > > > > > > Thanks. > > > > > > > > -- > > > > You received this message because you are subscribed to the > > > > ``` > > Google Groups "elasticsearch" group. > > > ``` > > > > To unsubscribe from this group and stop receiving emails > > > > ``` > > from it, send an email to > > > ``` > > > >elasticsearc...@googlegroups.com . > > > > For more options, visithttps:// > > > > ``` > > [groups.google.com/groups/opt\_out](http://groups.google.com/groups/opt_out) [http://groups.google.com/groups/opt\_out](http://groups.google.com/groups/opt_out) > \<[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > ``` > > >. > > > > > > -- > > > Costin > > > > > > -- > > > You received this message because you are subscribed to the Google > > > > ``` > > Groups "elasticsearch" group. > > > ``` > > > To unsubscribe from this group and stop receiving emails from it, > > > > ``` > > send an email to > > > ``` > > >elasticsearc...@googlegroups.com . > > > For more options, visithttps://groups.google.com/groups/opt_out < > > > > ``` > > [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)\>. > > > ``` > > -- > > Costin > > > > ``` > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \ \ > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \\>. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/3a82d348-066c-421e-996e-69b81455f175%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3a82d348-066c-421e-996e-69b81455f175%40googlegroups.com) > > > \< > > [https://groups.google.com/d/msgid/elasticsearch/3a82d348-066c-421e-996e-69b81455f175%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/3a82d348-066c-421e-996e-69b81455f175%40googlegroups.com?utm_medium=email&utm_source=footer)\>. > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/35f03b6d-de8a-4bd4-91e8-2fda7a940025%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/35f03b6d-de8a-4bd4-91e8-2fda7a940025%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:03am UTC](https://discuss.elastic.co/t/hadoop-pipelining-hadoop-spark-with-elasticsearch/14093/10 "2017-07-06T01:03:26Z")