# Cannot find mapping for spark/test-1463342115226 - one is required before using Spark SQL

**URL:** <https://discuss.elastic.co/t/cannot-find-mapping-for-spark-test-1463342115226-one-is-required-before-using-spark-sql/50088>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [May 15, 2016, 8:10pm UTC](https://discuss.elastic.co/t/cannot-find-mapping-for-spark-test-1463342115226-one-is-required-before-using-spark-sql/50088 "2016-05-15T20:10:33Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![chris\_snow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_snow/32/9363_2.png) [@chris\_snow](https://discuss.elastic.co/u/chris_snow)\
**Post date:** [May 15, 2016, 8:10pm UTC](https://discuss.elastic.co/t/cannot-find-mapping-for-spark-test-1463342115226-one-is-required-before-using-spark-sql/50088/1 "2016-05-15T20:10:33Z")

</div>

I'm trying to create a hello world type example that uses spark to save data into elasticsearch (using the es spark package 2.3.1) and then read it back again.

The example works ok on spark 1.5, but when I try to run it against spark 1.6, I hit the error in the subject line.

My code looks like this:

```
snowch$ cat export_to_elasticsearch.py

from __future__ import print_function

import sys
from operator import add

from pyspark import SparkContext
from pyspark import SparkConf
from pyspark.sql import SQLContext

if __name__ == " __main__":
    if len(sys.argv) != 6:
        print("Usage: export_to_elasticsearch.py <host> <port> <user> <pass> <path>", file=sys.stderr)
        exit(-1)

    host = sys.argv[1]
    port = sys.argv[2]
    user = sys.argv[3]
    password = sys.argv[4]
    path = sys.argv[5]

    conf = SparkConf().setAppName("Elasticsearch example")

    # see https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html
    conf.set("es.nodes",host)
    conf.set("es.port",str(port))
    conf.set("es.net.http.auth.user",user)
    conf.set("es.net.http.auth.pass",password)
    conf.set("es.net.ssl","true")
    conf.set("es.net.ssl.truststore.location","truststore.jks")
    conf.set("es.net.ssl.truststore.pass","mypassword")
    conf.set("es.nodes.wan.only","true")

    sc = SparkContext(conf=conf)
    sqlContext = SQLContext(sc)

    # create a test document
    d = [{'name': 'Alice', 'age': 1}]
    df = sqlContext.createDataFrame(d)

    # persist it to elasticsearch
    df.write.format("org.elasticsearch.spark.sql").save("spark/{0}".format(path))

    sc.stop()

```

and

```
snowch$ cat import_from_elasticsearch.py

from __future__ import print_function

import sys
from operator import add

from pyspark import SparkContext
from pyspark import SparkConf
from pyspark.sql import SQLContext

if __name__ == " __main__":
    if len(sys.argv) != 6:
        print("Usage: export_to_elasticsearch.py <host> <port> <user> <pass> <path>", file=sys.stderr)
        exit(-1)

    host = sys.argv[1]
    port = sys.argv[2]
    user = sys.argv[3]
    password = sys.argv[4]
    path = sys.argv[5]

    conf = SparkConf().setAppName("Elasticsearch example")

    # see https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html
    conf.set("es.nodes",host)
    conf.set("es.port",str(port))
    conf.set("es.net.http.auth.user",user)
    conf.set("es.net.http.auth.pass",password)
    conf.set("es.net.ssl","true")
    conf.set("es.net.ssl.truststore.location","truststore.jks")
    conf.set("es.net.ssl.truststore.pass","mypassword")
    conf.set("es.nodes.wan.only","true")

    sc = SparkContext(conf=conf)
    sqlContext = SQLContext(sc)

    # read the data from elasticsearch
    esdata = sqlContext.read.format("es").load("spark/{0}".format(path))

    print(esdata.rdd.take(10))

    sc.stop()
```

---

<div class="post-metadata">

**Author:** ![chris\_snow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_snow/32/9363_2.png) [@chris\_snow](https://discuss.elastic.co/u/chris_snow)\
**Post date:** [May 16, 2016, 8:01am UTC](https://discuss.elastic.co/t/cannot-find-mapping-for-spark-test-1463342115226-one-is-required-before-using-spark-sql/50088/2 "2016-05-16T08:01:17Z")

</div>

Ok, so the issue was because the type name had changed between test and I was trying to read from a type that didn't exist.

The error message threw me because it made me think the type did exist.

---

<div class="post-metadata">

**Author:** ![sumit\_gupta\_sgt](https://avatars.discourse-cdn.com/v4/letter/s/dec6dc/32.png) [@sumit\_gupta\_sgt](https://discuss.elastic.co/u/sumit_gupta_sgt)\
**Post date:** [August 22, 2016, 6:23pm UTC](https://discuss.elastic.co/t/cannot-find-mapping-for-spark-test-1463342115226-one-is-required-before-using-spark-sql/50088/3 "2016-08-22T18:23:52Z")

</div>

Hi Chris,

I am also trying same stuff with elasticsearch-spark\_2.11-2.3.3.jar with ES version 2.3.3. and Spark Version 2.0.0 ..My problem is i am able to read data but when i am trying to write back to ES it is throwing some Error:

Can you please tell me about what is the difference between using elasticsearch-spark\_2.11-2.3.3.jar and elasticsearch-hadoop-2.3.3.jar?

Here is the full Stack Trace:

py4j.protocol.Py4JJavaError: An error occurred while calling o34.save.  
: java.lang.AbstractMethodError: org.elasticsearch.spark.sql.DefaultSource.createRelation(Lorg/apache/spark/sql/SQLContext;Lorg/apache/spark/sql/SaveMode;Lscala/collection/immutable/Map;Lorg/apache/spark/sql/Dataset;)Lorg/apache/spark/sql/sources/BaseRelation;  
at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:429)  
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:211)  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)  
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
at java.lang.reflect.Method.invoke(Method.java:606)  
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)  
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)  
at py4j.Gateway.invoke(Gateway.java:280)  
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)  
at py4j.commands.CallCommand.execute(CallCommand.java:79)  
at py4j.GatewayConnection.run(GatewayConnection.java:211)  
at java.lang.Thread.run(Thread.java:745)

Thanks,  
Sumit

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [August 22, 2016, 9:11pm UTC](https://discuss.elastic.co/t/cannot-find-mapping-for-spark-test-1463342115226-one-is-required-before-using-spark-sql/50088/4 "2016-08-22T21:11:31Z")

</div>

> [@sumit\_gupta\_sgt](#):
>
> Can you please tell me about what is the difference between using elasticsearch-spark\_2.11-2.3.3.jar and elasticsearch-hadoop-2.3.3.jar?

Since Scala 2.11 is not backwards compatible with Scala 2.10, the spark portion of the library is also released as a standalone jar with versions that are compatible with each Scala version.

---

<div class="post-metadata">

**Author:** ![sumit\_gupta\_sgt](https://avatars.discourse-cdn.com/v4/letter/s/dec6dc/32.png) [@sumit\_gupta\_sgt](https://discuss.elastic.co/u/sumit_gupta_sgt)\
**Post date:** [August 23, 2016, 4:55am UTC](https://discuss.elastic.co/t/cannot-find-mapping-for-spark-test-1463342115226-one-is-required-before-using-spark-sql/50088/5 "2016-08-23T04:55:37Z")

</div>

> [@james.baiera](#):
>
> cala 2.11 is not backwar

@james.baiera Thanks for your quick reply.  
So you mean to say can i directly use elasticsearch-spark\_2.11-2.3.3.jar this jar for writing in ES indexs.

Thanks  
Sumit

---

<div class="post-metadata">

**Author:** ![sumit\_gupta\_sgt](https://avatars.discourse-cdn.com/v4/letter/s/dec6dc/32.png) [@sumit\_gupta\_sgt](https://discuss.elastic.co/u/sumit_gupta_sgt)\
**Post date:** [August 23, 2016, 5:29am UTC](https://discuss.elastic.co/t/cannot-find-mapping-for-spark-test-1463342115226-one-is-required-before-using-spark-sql/50088/6 "2016-08-23T05:29:29Z")

</div>

> [@chris\_snow](#):
>
> e that didn't exist.

@james.baiera Can you please let me know when i am trying to write ES indexes it is throwing error.  
in the below conversation i already attached stack-trace.

Thanks,  
Sumit

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [August 23, 2016, 2:20pm UTC](https://discuss.elastic.co/t/cannot-find-mapping-for-spark-test-1463342115226-one-is-required-before-using-spark-sql/50088/7 "2016-08-23T14:20:21Z")

</div>

ES Hadoop does not support Spark 2.0 until version 5.0.0-alpha5. You'll need to download that version to interact with Spark 2.0.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:23pm UTC](https://discuss.elastic.co/t/cannot-find-mapping-for-spark-test-1463342115226-one-is-required-before-using-spark-sql/50088/8 "2017-07-06T13:23:35Z")

</div>