# Nested fields upsert with Spark not working

**URL:** <https://discuss.elastic.co/t/nested-fields-upsert-with-spark-not-working/146633>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [August 30, 2018, 7:30am UTC](https://discuss.elastic.co/t/nested-fields-upsert-with-spark-not-working/146633 "2018-08-30T07:30:58Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![cluengo](https://avatars.discourse-cdn.com/v4/letter/c/d9b06d/32.png) [@cluengo](https://discuss.elastic.co/u/cluengo)\
**Post date:** [August 30, 2018, 7:30am UTC](https://discuss.elastic.co/t/nested-fields-upsert-with-spark-not-working/146633/1 "2018-08-30T07:30:58Z")

</div>

Hi,

I already asked this a few months ago, but since the answer was that it would be fixed in a following release, I wanted to know whether I'm doing something wrong, or if it's still not fixed.

I'm trying to update a nested field using Spark and a scripted update. The code I'm using is:

update\_params = "new\_samples: samples"  
update\_script = "ctx.\_source.samples += new\_samples"

es\_conf = {  
"es.mapping.id": "id",  
"es.mapping.exclude": "id",  
"es.write.operation": "upsert",  
"es.update.script.params": update\_params,  
"es.update.script.inline": update\_script  
}

result.write.format("org.elasticsearch.spark.sql").options(\*\*es\_conf).option("es.nodes",configuration["elasticsearch"]["host"]).option("es.port",configuration["elasticsearch"]["port"] ).save(configuration["elasticsearch"]["index\_name"]+"/"+configuration["version"],mode='append')

And the schema of the field is:

|-- samples: array (nullable = true)  
| |-- element: struct (containsNull = true)  
| | |-- gq: integer (nullable = true)  
| | |-- dp: integer (nullable = true)  
| | |-- gt: string (nullable = true)  
| | |-- adBug: array (nullable = true)  
| | | |-- element: integer (containsNull = true)  
| | |-- ad: double (nullable = true)  
| | |-- sample: string (nullable = true)

And I get the following error:

py4j.protocol.Py4JJavaError: An error occurred while calling o83.save.  
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1, localhost, executor driver): java.lang.ClassCastException: scala.collection.mutable.WrappedArray$ofRef cannot be cast to scala.Tuple2

Thanks so much!

---

<div class="post-metadata">

**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\
**Post date:** [September 4, 2018, 5:31pm UTC](https://discuss.elastic.co/t/nested-fields-upsert-with-spark-not-working/146633/2 "2018-09-04T17:31:56Z")

</div>

Hello, whats is the index mapping?

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [September 4, 2018, 6:08pm UTC](https://discuss.elastic.co/t/nested-fields-upsert-with-spark-not-working/146633/3 "2018-09-04T18:08:23Z")

</div>

This is definitely a bug. Replied to [https://github.com/elastic/elasticsearch-hadoop/issues/1190](https://github.com/elastic/elasticsearch-hadoop/issues/1190)

---

<div class="post-metadata">

**Author:** ![cluengo](https://avatars.discourse-cdn.com/v4/letter/c/d9b06d/32.png) [@cluengo](https://discuss.elastic.co/u/cluengo)\
**Post date:** [September 5, 2018, 8:11am UTC](https://discuss.elastic.co/t/nested-fields-upsert-with-spark-not-working/146633/4 "2018-09-05T08:11:33Z")

</div>

Ok, thanks so much!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 3, 2018, 8:11am UTC](https://discuss.elastic.co/t/nested-fields-upsert-with-spark-not-working/146633/5 "2018-10-03T08:11:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.