I am running a Zeppelin 0.7.1 instance and I want to use the Spark connector. I added the dependency
org.elasticsearch:elasticsearch-spark-20_2.10:5.4.1
And yet I get the following error message everytime I run any code on the spark interpreter (even if it is a random import statement or a plain sc.version):
java.lang.NullPointerException
at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:38)
at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:33)
at org.apache.zeppelin.spark.SparkInterpreter.createSparkContext_2(SparkInterpreter.java:391)
at org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:380)
at org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:146)
at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:828)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:483)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
This is strictly related to this dependency. If I remove the dependency the interpreter works flawlessly. I also tested it for scala version 11 (so -20_2.11). The same error happens, so it is not a scala incompatibility.
I also added the following entries in the interpreter:
spark.es.nodes localhost
spark.es.port 9200
spark.es.index.auto.create true
Is there some other configuration that I haven't added yet? Any idea why this NullPointerException happens?
Edit: Some clarifications. When I run a local version of Zeppelin and a standalone Spark cluster in my localmachine, I do not get any errors. I can use the Elastic dependency. However, I need to use another Zeppelin that is hosted on another machine. Only through that Zeppelin can I make use of a big proper spark cluster for running heavy jobs. Yet the same settings do not work for the big cluster and I get the error above.
Edit 2: I finally found out where the incompatibility lies: The elasticsearch-spark connector clashes for some reason with my play json dependency:
com.typesafe.play:play-json_2.11:2.5.15
If I get rid of this dependency (which I need actually) then the es-spark connection works fine. However I cannot get any further. I don't know how to find out which packets/dependencies of play.json and es-spark are actually clashing and causing this.