# Spark SQL types are not handled through basic RDD saveToEs() calls

**URL:** <https://discuss.elastic.co/t/spark-sql-types-are-not-handled-through-basic-rdd-savetoes-calls/66557>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [November 18, 2016, 3:38pm UTC](https://discuss.elastic.co/t/spark-sql-types-are-not-handled-through-basic-rdd-savetoes-calls/66557 "2016-11-18T15:38:02Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![jspooner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jspooner/32/12984_2.png) [@jspooner](https://discuss.elastic.co/u/jspooner)\
**Post date:** [November 18, 2016, 3:38pm UTC](https://discuss.elastic.co/t/spark-sql-types-are-not-handled-through-basic-rdd-savetoes-calls/66557/1 "2016-11-18T15:38:02Z")

</div>

Given the following Dataset

```
aDF
res33: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [device_id: string]

aDF.rdd
res34: org.apache.spark.rdd.RDD[org.apache.spark.sql.Row] = MapPartitionsRDD[129] at rdd at <console>:67

aDF.show()

+-------------------------------------------+                                   
|device_id |
+-------------------------------------------+
|g.40 |
|a.E2 |
|a.39 |

```

I would expect the ability to save it to ES with the following command.

```
EsSpark.saveToEs(aDF.rdd, "production-devices-spark-shell/device")

16/11/18 15:28:22 WARN TaskSetManager: Lost task 0.0 in stage 86.0 (TID 3595, ip-10-1-181-210.aws.vrv): org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Spark SQL types are not handled through basic RDD saveToEs() calls; typically this is a mistake(as the SQL schema will be ignored). Use 'org.elasticsearch.spark.sql' package instead
	at org.elasticsearch.spark.serialization.ScalaValueWriter.org$elasticsearch$spark$serialization$ScalaValueWriter$$doWrite(ScalaValueWriter.scala:111)
	at org.elasticsearch.spark.serialization.ScalaValueWriter.write(ScalaValueWriter.scala:37)
	at org.elasticsearch.hadoop.serialization.builder.ContentBuilder.value(ContentBuilder.java:53)
	at org.elasticsearch.hadoop.serialization.bulk.TemplatedBulk.doWriteObject(TemplatedBulk.java:71)
	at org.elasticsearch.hadoop.serialization.bulk.TemplatedBulk.write(TemplatedBulk.java:58)
	at org.elasticsearch.hadoop.rest.RestRepository.writeToIndex(RestRepository.java:159)
	at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:67)
	at org.elasticsearch.spark.rdd.EsSpark$$anonfun$doSaveToEs$1.apply(EsSpark.scala:102)
	at org.elasticsearch.spark.rdd.EsSpark$$anonfun$doSaveToEs$1.apply(EsSpark.scala:102)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
	at org.apache.spark.scheduler.Task.run(Task.scala:86)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

```

The error message says `Use 'org.elasticsearch.spark.sql' package instead` but what exactly does that mean?

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [November 18, 2016, 5:30pm UTC](https://discuss.elastic.co/t/spark-sql-types-are-not-handled-through-basic-rdd-savetoes-calls/66557/2 "2016-11-18T17:30:13Z")

</div>

> [@jspooner](#):
>
> The error message says Use 'org.elasticsearch.spark.sql' package instead but what exactly does that mean?

We provide native support for Spark SQL DataFrames and Datasets in ES-Hadoop. These often require some different logic to handle the serialization correctly. Please take a look at our [documentation section about SparkSQL](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/spark.html#spark-sql) for the correct API calls.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 16, 2016, 5:30pm UTC](https://discuss.elastic.co/t/spark-sql-types-are-not-handled-through-basic-rdd-savetoes-calls/66557/3 "2016-12-16T17:30:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.