Is there a way to "update" ES records using Spark?

Elastic Stack Elasticsearch
1

I am using ES version 1.3.2, and Spark 1.1.0.
I can successfully read and write records from/to ES using newAPIHadoopRDD()
and saveAsNewAPIHadoopDataset().
However, I am struggling to find a way to update records. Even I specify a
'key' in ESOutputFormat it gets ignored, as documented clearly.
So my question is : Is there a way to specify document ID and custom
routing values when writing to ES using Spark? If yes, how?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/529fb146-bea8-48ac-aed0-d6908775f85d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Anyone has an idea?
At least if I get to know whether this is possible or not, that'll be a
great help.

Thanks.

On Wednesday, October 1, 2014 3:46:51 PM UTC+5:30, Preeti Raj - Buchhada
wrote:

I am using ES version 1.3.2, and Spark 1.1.0.
I can successfully read and write records from/to ES using newAPIHadoopRDD()
and saveAsNewAPIHadoopDataset().
However, I am struggling to find a way to update records. Even I specify a
'key' in ESOutputFormat it gets ignored, as documented clearly.
So my question is : Is there a way to specify document ID and custom
routing values when writing to ES using Spark? If yes, how?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b6e4628a-5106-4f2b-997d-e790a8aeb455%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

You can the mapping options [1], namely es.mapping.id to specify the
id field of your documents.

[1] Elasticsearch Platform — Find real-time answers at scale | Elastic

On Mon, Oct 13, 2014 at 12:55 PM, Preeti Raj - Buchhada
pbuchhada@gmail.com wrote:

Anyone has an idea?
At least if I get to know whether this is possible or not, that'll be a
great help.

Thanks.

On Wednesday, October 1, 2014 3:46:51 PM UTC+5:30, Preeti Raj - Buchhada
wrote:

I am using ES version 1.3.2, and Spark 1.1.0.
I can successfully read and write records from/to ES using
newAPIHadoopRDD() and saveAsNewAPIHadoopDataset().
However, I am struggling to find a way to update records. Even I specify a
'key' in ESOutputFormat it gets ignored, as documented clearly.
So my question is : Is there a way to specify document ID and custom
routing values when writing to ES using Spark? If yes, how?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b6e4628a-5106-4f2b-997d-e790a8aeb455%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJogdmc6Ks1zfaq7%3DssO%3DQ4rt0ddP3K2e_H7ebwPm%2B3gRsBY6w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

4

Thanks for your reply Costin.
However, we have a need to compute a custom ID based on concatenation of
multiple fields values and then computing the hash value. So simply
specifying 'es.mapping.id' will not help in our case.

Is there any other way?

On Monday, October 13, 2014 4:08:05 PM UTC+5:30, Costin Leau wrote:

You can the mapping options [1], namely es.mapping.id to specify the
id field of your documents.

[1]
Elasticsearch Platform — Find real-time answers at scale | Elastic

On Mon, Oct 13, 2014 at 12:55 PM, Preeti Raj - Buchhada
<pbuc...@gmail.com <javascript:>> wrote:

Anyone has an idea?
At least if I get to know whether this is possible or not, that'll be a
great help.

Thanks.

On Wednesday, October 1, 2014 3:46:51 PM UTC+5:30, Preeti Raj - Buchhada
wrote:

I am using ES version 1.3.2, and Spark 1.1.0.
I can successfully read and write records from/to ES using
newAPIHadoopRDD() and saveAsNewAPIHadoopDataset().
However, I am struggling to find a way to update records. Even I
specify a
'key' in ESOutputFormat it gets ignored, as documented clearly.
So my question is : Is there a way to specify document ID and custom
routing values when writing to ES using Spark? If yes, how?

--
You received this message because you are subscribed to the Google
Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit

https://groups.google.com/d/msgid/elasticsearch/b6e4628a-5106-4f2b-997d-e790a8aeb455%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fc10e3fb-158b-4ae2-9117-beea8a620865%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.