# Cannot connect Spark to Elasticsearch \*RESOLVED\*

**URL:** <https://discuss.elastic.co/t/cannot-connect-spark-to-elasticsearch-resolved/29811>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [September 22, 2015, 8:43pm UTC](https://discuss.elastic.co/t/cannot-connect-spark-to-elasticsearch-resolved/29811 "2015-09-22T20:43:45Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![bjoernsteffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bjoernsteffens/32/545_2.png) [@bjoernsteffens](https://discuss.elastic.co/u/bjoernsteffens)\
**Post date:** [September 22, 2015, 8:43pm UTC](https://discuss.elastic.co/t/cannot-connect-spark-to-elasticsearch-resolved/29811/1 "2015-09-22T20:43:45Z")

</div>

Hi,

I am trying to connect my Spark to Elasticsearch to read log data. I get all sorts of weird error messages. I would be happy of someone could give me a pointer with some complete documentation.

spark-shell --master mesos://spark-m00:5050 --jars /images/spark\_jars/elasticsearch-hadoop-2.1.1.jar --conf spark.es.nodes=sidr35dgapan001 --conf spark.es.port=9200 --conf spark.es.nodes.discovery=false --conf spark.es.http.timeout=5m --conf spark.es.resource=logstash-2015.09.22/bjoern

import org.apache.spark.SparkContext  
import org.apache.spark.SparkContext.\_  
import org.elasticsearch.spark.rdd.EsSpark  
import org.elasticsearch.spark.\_  
import org.elasticsearch.hadoop.util.ObjectUtils

val bjoernRDD = sc.esRDD("logstash-2015.09.22/bjoern")

java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;  
at org.elasticsearch.spark.rdd.EsSpark$.esRDD(EsSpark.scala:26)  
at org.elasticsearch.spark.package$SparkContextFunctions.esRDD(package.scala:20)  
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:30)  
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:35)  
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:37)  
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:39)  
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:41)  
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:43)  
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:45)  
at $iwC$$iwC$$iwC$$iwC$$iwC.(:47)  
at $iwC$$iwC$$iwC$$iwC.(:49)  
at $iwC$$iwC$$iwC.(:51)  
at $iwC$$iwC.(:53)  
at $iwC.(:55)

---

<div class="post-metadata">

**Author:** ![bjoernsteffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bjoernsteffens/32/545_2.png) [@bjoernsteffens](https://discuss.elastic.co/u/bjoernsteffens)\
**Post date:** [September 23, 2015, 3:44pm UTC](https://discuss.elastic.co/t/cannot-connect-spark-to-elasticsearch-resolved/29811/2 "2015-09-23T15:44:27Z")

</div>

BTW here are the versions I am on

Spark: 1.5 (spark-1.5.0-bin-hadoop2.6)  
ES: 1.7.2

---

<div class="post-metadata">

**Author:** ![bjoernsteffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bjoernsteffens/32/545_2.png) [@bjoernsteffens](https://discuss.elastic.co/u/bjoernsteffens)\
**Post date:** [September 23, 2015, 7:06pm UTC](https://discuss.elastic.co/t/cannot-connect-spark-to-elasticsearch-resolved/29811/3 "2015-09-23T19:06:31Z")

</div>

#Resolved

# 

After a lot of reading and failing I have a Spark configuration that connects and reads from ES. What confused me when downloading the jars was there there is a whole bunch of them but you only need one. Adding some comments here in case someone should run into the same issue.

####elasticsearch-hadoop-2.1.1.jar

# 

In addition I had a bit of junk in my root folder which I had to clean out. These were the steps that got my Spark V1.5 connected with ES V1.7.2

####execute the following commands as the root user

# 

cd /  
rm -rf .ivy2

export CLASSPATH=/images/spark\_jars/elasticsearch-hadoop-2.1.1.jar

spark-shell --master local[4] --jars /images/spark\_jars/elasticsearch-hadoop-2.1.1.jar --conf spark.es.nodes=spark-m00 --conf spark.es.port=9200 --conf spark.es.nodes.discovery=false --conf spark.es.http.timeout=5m --conf spark.es.resource=logstash-2015.09.22/bjoern

####When the Spark shell is available enter these Scala lines

import org.apache.spark.sql.SQLContext  
import org.elasticsearch.spark  
import org.elasticsearch.spark.sql  
import org.elasticsearch.hadoop

val df = sqlContext.read.format("org.elasticsearch.spark.sql").load("logstash-2015.09.23/bjoern")  
df.printSchema()  
df.show()

#### The results is here and I can massage it accordingly.

# 

+--------------------+--------+---------------+-----+--------------------+-----------------+------+------+  
| @timestamp|@version| file|geoip| host| message|offset| type|  
+--------------------+--------+---------------+-----+--------------------+-----------------+------+------+  
|2015-09-23 13:57:...| 1|/tmp/bjoern.log| null|spark-c01|spark driver main| 51876|bjoern|  
|2015-09-23 13:57:...| 1|/tmp/bjoern.log| null|spark-c01|spark driver main| 51966|bjoern|  
|2015-09-23 13:57:...| 1|/tmp/bjoern.log| null|spark-c01|spark driver main| 52020|bjoern|  
|2015-09-23 13:57:...| 1|/tmp/bjoern.log| null|spark-c01|spark driver main| 52038|bjoern|  
|2015-09-23 13:57:...| 1|/tmp/bjoern.log| null|spark-c01|spark driver main| 52092|bjoern|  
|2015-09-23 13:57:...| 1|/tmp/bjoern.log| null|spark-c01|spark driver main| 52182|bjoern|  
|2015-09-23 13:57:...| 1|/tmp/bjoern.log| null|spark-c01|spark driver main| 52272|bjoern|  
|2015-09-23 13:57:...| 1|/tmp/bjoern.log| null|spark-c01|spark driver main| 52362|bjoern|  
|2015-09-23 13:57:...| 1|/tmp/bjoern.log| null|spark-c01|spark driver main| 52452|bjoern|  
|2015-09-23 13:57:...| 1|/tmp/bjoern.log| null|spark-c01|spark driver main| 52524|bjoern|  
|2015-09-23 13:57:...| 1|/tmp/bjoern.log| null|spark-c01|spark driver main| 52614|bjoern|  
|2015-09-23 13:57:...| 1|/tmp/bjoern.log| null|spark-c01|spark driver main| 52704|bjoern|  
|2015-09-23 13:57:...| 1|/tmp/bjoern.log| null|spark-c01|spark driver main| 52794|bjoern|  
|2015-09-23 13:57:...| 1|/tmp/bjoern.log| null|spark-c01|spark driver main| 52884|bjoern|  
+--------------------+--------+---------------+-----+--------------------+-----------------+------+------+

---

<div class="post-metadata">

**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\
**Post date:** [September 26, 2015, 2:25pm UTC](https://discuss.elastic.co/t/cannot-connect-spark-to-elasticsearch-resolved/29811/4 "2015-09-26T14:25:02Z")

</div>

Glad to hear you sorted things out. For best results, Spark 1.5 is supported currently in the [dev builds][1] , namely the upcoming 2.1.2 and 2.2.0.m2.

Cheers,  
[1]: [https://www.elastic.co/guide/en/elasticsearch/hadoop/current/install.html#download-dev](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/install.html#download-dev)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:27pm UTC](https://discuss.elastic.co/t/cannot-connect-spark-to-elasticsearch-resolved/29811/5 "2017-07-06T13:27:29Z")

</div>