# Zeppelin and Elasticsearch-Spark incompatibility

**URL:** <https://discuss.elastic.co/t/zeppelin-and-elasticsearch-spark-incompatibility/88935>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [June 10, 2017, 10:13pm UTC](https://discuss.elastic.co/t/zeppelin-and-elasticsearch-spark-incompatibility/88935 "2017-06-10T22:13:14Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![dona](https://avatars.discourse-cdn.com/v4/letter/d/7c8e57/32.png) [@dona](https://discuss.elastic.co/u/dona)\
**Post date:** [June 10, 2017, 10:13pm UTC](https://discuss.elastic.co/t/zeppelin-and-elasticsearch-spark-incompatibility/88935/1 "2017-06-10T22:13:14Z")

</div>

I am running a Zeppelin 0.7.1 instance and I want to use the Spark connector. I added the dependency

```
org.elasticsearch:elasticsearch-spark-20_2.10:5.4.1	

```

And yet I get the following error message everytime I run any code on the spark interpreter (even if it is a random import statement or a plain sc.version):

```
java.lang.NullPointerException
	at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:38)
	at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:33)
	at org.apache.zeppelin.spark.SparkInterpreter.createSparkContext_2(SparkInterpreter.java:391)
	at org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:380)
	at org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:146)
	at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:828)
	at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:483)
	at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
	at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

```

This is strictly related to this dependency. If I remove the dependency the interpreter works flawlessly. I also tested it for scala version 11 (so -20\_2.11). The same error happens, so it is not a scala incompatibility.

I also added the following entries in the interpreter:

```
spark.es.nodes	localhost
spark.es.port	9200
spark.es.index.auto.create	true

```

Is there some other configuration that I haven't added yet? Any idea why this NullPointerException happens?

**Edit** : Some clarifications. When I run a local version of Zeppelin and a standalone Spark cluster in my localmachine, I do not get any errors. I can use the Elastic dependency. However, I need to use another Zeppelin that is hosted on another machine. Only through that Zeppelin can I make use of a big proper spark cluster for running heavy jobs. Yet the same settings do not work for the big cluster and I get the error above.

**Edit 2** : I finally found out where the incompatibility lies: The elasticsearch-spark connector clashes for some reason with my play json dependency:

`com.typesafe.play:play-json_2.11:2.5.15`

If I get rid of this dependency (which I need actually) then the es-spark connection works fine. However I cannot get any further. I don't know how to find out which packets/dependencies of play.json and es-spark are actually clashing and causing this.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 8, 2017, 10:18pm UTC](https://discuss.elastic.co/t/zeppelin-and-elasticsearch-spark-incompatibility/88935/2 "2017-07-08T22:18:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.