# Spark to ES NoSuchMethodError - DataFrameFieldExtractor.extractField

**URL:** <https://discuss.elastic.co/t/spark-to-es-nosuchmethoderror-dataframefieldextractor-extractfield/96042>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [August 7, 2017, 5:06am UTC](https://discuss.elastic.co/t/spark-to-es-nosuchmethoderror-dataframefieldextractor-extractfield/96042 "2017-08-07T05:06:56Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![dragick](https://avatars.discourse-cdn.com/v4/letter/d/53a042/32.png) [@dragick](https://discuss.elastic.co/u/dragick)\
**Post date:** [August 7, 2017, 5:06am UTC](https://discuss.elastic.co/t/spark-to-es-nosuchmethoderror-dataframefieldextractor-extractfield/96042/1 "2017-08-07T05:06:57Z")

</div>

Hello! I am trying to figure out how to upsert rows to ElasticSearch from Spark.  
I am using elasticsearch-spark-20\_2.10-5.5.1.jar, on a Spark cluster running Spark2.1 with Scala 2.1

My DataFrame printSchema() looks like this;

```
root
 |-- datasetid: integer (nullable = false)
 |-- name: string (nullable = false)

```

The code I am using to write the DataFrame looks like this;

```
var ElasticSearchHost = "http://url-to-elasticsearch.com"
var esConfig:Map[String,String] = Map("es.nodes" -> ElasticSearchHost,
                                      "es.port" -> "443",
                                      "es.nodes.wan.only" -> "true",
                                      "es.net.ssl" -> "true",
                                      "es.nodes.wan.only" -> "true",
                                      "es.mapping.id" -> "datasetid",
                                      "es.write.operation" -> "upsert")

dfSet1.write.format("org.elasticsearch.spark.sql")
        .options(esConfig)
        .save("indexofstuff/persons")

```

This is returning the following error;

```
org.apache.spark.SparkException: Job aborted due to stage failure: Task 28 in stage 87.0 failed 4 times, most recent failure: Lost task 28.3 in stage 87.0 (TID 3943, 10.164.225.89, executor 0): java.lang.NoSuchMethodError: scala.runtime.ObjectRef.create(Ljava/lang/Object;)Lscala/runtime/ObjectRef;
	at org.elasticsearch.spark.sql.DataFrameFieldExtractor.extractField(DataFrameFieldExtractor.scala:29)
	at org.elasticsearch.hadoop.serialization.field.ConstantFieldExtractor.field(ConstantFieldExtractor.java:36)
	at org.elasticsearch.hadoop.serialization.bulk.AbstractBulkFactory$FieldWriter.write(AbstractBulkFactory.java:94)
	at org.elasticsearch.hadoop.serialization.bulk.TemplatedBulk.writeTemplate(TemplatedBulk.java:80)
	at org.elasticsearch.hadoop.serialization.bulk.TemplatedBulk.write(TemplatedBulk.java:56)
	at org.elasticsearch.hadoop.rest.RestRepository.writeToIndex(RestRepository.java:159)
	at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:67)
	at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:94)
	at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:94)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:99)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

```

How do I pass in the ID field of my DataFrame? If I take out [es.mapping.id](http://es.mapping.id) and es.write.operation it writes without issue, BUT this causes ElasticSearch to generate the ID which makes me unable to update entries.

Any further information I can provide, please let me know and I'll get right on it.

---

<div class="post-metadata">

**Author:** ![eliasah](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eliasah/32/34741_2.png) [@eliasah](https://discuss.elastic.co/u/eliasah)\
**Post date:** [August 7, 2017, 8:24am UTC](https://discuss.elastic.co/t/spark-to-es-nosuchmethoderror-dataframefieldextractor-extractfield/96042/2 "2017-08-07T08:24:30Z")

</div>

I'd definitely check that the version of the ES cluster, Spark cluster, JDK & Scala versions along with the dependencies that you are using in your project. They don't seem to be compatible.

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [August 7, 2017, 4:12pm UTC](https://discuss.elastic.co/t/spark-to-es-nosuchmethoderror-dataframefieldextractor-extractfield/96042/3 "2017-08-07T16:12:53Z")

</div>

> [@dragick](#):
>
> Spark cluster running Spark2.1 with **Scala 2.1**

Do you mean Scala **2.10** or **2.11**? This error almost always indicates that you're using a 2.10 artifact with a 2.11 Scala distribution.

---

<div class="post-metadata">

**Author:** ![dragick](https://avatars.discourse-cdn.com/v4/letter/d/53a042/32.png) [@dragick](https://discuss.elastic.co/u/dragick)\
**Post date:** [August 8, 2017, 1:24am UTC](https://discuss.elastic.co/t/spark-to-es-nosuchmethoderror-dataframefieldextractor-extractfield/96042/4 "2017-08-08T01:24:41Z")

</div>

Silly me, version mismatch. I should have checked into that.  
Definately running Scala 2.1, not 2.11

Now using elasticsearch-spark-20\_2.10-5.5.1.jar and it's going smoothly

Thanks for the assist!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 5, 2017, 1:25am UTC](https://discuss.elastic.co/t/spark-to-es-nosuchmethoderror-dataframefieldextractor-extractfield/96042/5 "2017-09-05T01:25:03Z")

</div>

Closed on September 5, 2017, 1:25am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.