# Is there a way to "update" ES records using Spark? **URL:** \ **Category:** Elasticsearch\ **Created:** [October 1, 2014, 10:16am UTC](https://discuss.elastic.co/t/is-there-a-way-to-update-es-records-using-spark/20021 "2014-10-01T10:16:51Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Preeti\_Raj\_Buchhada](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/preeti_raj_buchhada/32/8971_2.png) [@Preeti\_Raj\_Buchhada](https://discuss.elastic.co/u/Preeti_Raj_Buchhada)\ **Post date:** [October 1, 2014, 10:16am UTC](https://discuss.elastic.co/t/is-there-a-way-to-update-es-records-using-spark/20021/1 "2014-10-01T10:16:51Z")
I am using ES version 1.3.2, and Spark 1.1.0. I can successfully read and write records from/to ES using newAPIHadoopRDD() and saveAsNewAPIHadoopDataset(). However, I am struggling to find a way to update records. Even I specify a 'key' in ESOutputFormat it gets ignored, as documented clearly. So my question is : Is there a way to specify document ID and custom routing values when writing to ES using Spark? If yes, how? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/529fb146-bea8-48ac-aed0-d6908775f85d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/529fb146-bea8-48ac-aed0-d6908775f85d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Preeti\_Raj\_Buchhada](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/preeti_raj_buchhada/32/8971_2.png) [@Preeti\_Raj\_Buchhada](https://discuss.elastic.co/u/Preeti_Raj_Buchhada)\ **Post date:** [October 13, 2014, 9:55am UTC](https://discuss.elastic.co/t/is-there-a-way-to-update-es-records-using-spark/20021/2 "2014-10-13T09:55:25Z")
Anyone has an idea? At least if I get to know whether this is possible or not, that'll be a great help. Thanks. On Wednesday, October 1, 2014 3:46:51 PM UTC+5:30, Preeti Raj - Buchhada wrote: > I am using ES version 1.3.2, and Spark 1.1.0. > I can successfully read and write records from/to ES using newAPIHadoopRDD() > and saveAsNewAPIHadoopDataset(). > However, I am struggling to find a way to update records. Even I specify a > 'key' in ESOutputFormat it gets ignored, as documented clearly. > So my question is : Is there a way to specify document ID and custom > routing values when writing to ES using Spark? If yes, how? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b6e4628a-5106-4f2b-997d-e790a8aeb455%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b6e4628a-5106-4f2b-997d-e790a8aeb455%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [October 13, 2014, 10:37am UTC](https://discuss.elastic.co/t/is-there-a-way-to-update-es-records-using-spark/20021/3 "2014-10-13T10:37:59Z")
You can the mapping options [1], namely `es.mapping.id` to specify the id field of your documents. [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/master/configuration.html#cfg-mapping) On Mon, Oct 13, 2014 at 12:55 PM, Preeti Raj - Buchhada [pbuchhada@gmail.com](mailto:pbuchhada@gmail.com) wrote: > Anyone has an idea? > At least if I get to know whether this is possible or not, that'll be a > great help. > > Thanks. > > On Wednesday, October 1, 2014 3:46:51 PM UTC+5:30, Preeti Raj - Buchhada > wrote: > > > I am using ES version 1.3.2, and Spark 1.1.0. > > I can successfully read and write records from/to ES using > > newAPIHadoopRDD() and saveAsNewAPIHadoopDataset(). > > However, I am struggling to find a way to update records. Even I specify a > > 'key' in ESOutputFormat it gets ignored, as documented clearly. > > So my question is : Is there a way to specify document ID and custom > > routing values when writing to ES using Spark? If yes, how? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/b6e4628a-5106-4f2b-997d-e790a8aeb455%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b6e4628a-5106-4f2b-997d-e790a8aeb455%40googlegroups.com). > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJogdmc6Ks1zfaq7%3DssO%3DQ4rt0ddP3K2e\_H7ebwPm%2B3gRsBY6w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJogdmc6Ks1zfaq7%3DssO%3DQ4rt0ddP3K2e_H7ebwPm%2B3gRsBY6w%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Preeti\_Raj\_Buchhada](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/preeti_raj_buchhada/32/8971_2.png) [@Preeti\_Raj\_Buchhada](https://discuss.elastic.co/u/Preeti_Raj_Buchhada)\ **Post date:** [October 14, 2014, 4:44am UTC](https://discuss.elastic.co/t/is-there-a-way-to-update-es-records-using-spark/20021/4 "2014-10-14T04:44:46Z")
Thanks for your reply Costin. However, we have a need to compute a custom ID based on concatenation of multiple fields values and then computing the hash value. So simply specifying 'es.mapping.id' will not help in our case. Is there any other way? On Monday, October 13, 2014 4:08:05 PM UTC+5:30, Costin Leau wrote: > You can the mapping options [1], namely `es.mapping.id` to specify the > id field of your documents. > > [1] > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/master/configuration.html#cfg-mapping) > > On Mon, Oct 13, 2014 at 12:55 PM, Preeti Raj - Buchhada > \<[pbuc...@gmail.com](mailto:pbuc...@gmail.com) \\> wrote: > > > Anyone has an idea? > > At least if I get to know whether this is possible or not, that'll be a > > great help. > > > > Thanks. > > > > On Wednesday, October 1, 2014 3:46:51 PM UTC+5:30, Preeti Raj - Buchhada > > wrote: > > > > > I am using ES version 1.3.2, and Spark 1.1.0. > > > I can successfully read and write records from/to ES using > > > newAPIHadoopRDD() and saveAsNewAPIHadoopDataset(). > > > However, I am struggling to find a way to update records. Even I > > > specify a > > > 'key' in ESOutputFormat it gets ignored, as documented clearly. > > > So my question is : Is there a way to specify document ID and custom > > > routing values when writing to ES using Spark? If yes, how? > > > > -- > > You received this message because you are subscribed to the Google > > Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/b6e4628a-5106-4f2b-997d-e790a8aeb455%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b6e4628a-5106-4f2b-997d-e790a8aeb455%40googlegroups.com). > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/fc10e3fb-158b-4ae2-9117-beea8a620865%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fc10e3fb-158b-4ae2-9117-beea8a620865%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:56am UTC](https://discuss.elastic.co/t/is-there-a-way-to-update-es-records-using-spark/20021/5 "2017-07-06T00:56:27Z")