# How to run ES-Hadoop in Jupyter Notebook (Python or Scala)

**URL:** <https://discuss.elastic.co/t/how-to-run-es-hadoop-in-jupyter-notebook-python-or-scala/131456>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [May 11, 2018, 10:24am UTC](https://discuss.elastic.co/t/how-to-run-es-hadoop-in-jupyter-notebook-python-or-scala/131456 "2018-05-11T10:24:23Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Luan\_Ha\_Thanh](https://avatars.discourse-cdn.com/v4/letter/l/848f3c/32.png) [@Luan\_Ha\_Thanh](https://discuss.elastic.co/u/Luan_Ha_Thanh)\
**Post date:** [May 11, 2018, 10:24am UTC](https://discuss.elastic.co/t/how-to-run-es-hadoop-in-jupyter-notebook-python-or-scala/131456/1 "2018-05-11T10:24:24Z")

</div>

Hello there, I 'm following this tut to connect Elasticsearch and Spark:

> **[GitHub - IBM/elasticsearch-spark-recommender: Use Jupyter Notebooks to...](https://github.com/IBM/elasticsearch-spark-recommender)**
>
> Use Jupyter Notebooks to demonstrate how to build a Recommender with Apache Spark & Elasticsearch - GitHub - IBM/elasticsearch-spark-recommender: Use Jupyter Notebooks to demonstrate how to bui...

I use command:

> setx PYSPARK\_DRIVER\_PYTHON ipython  
> setx PYSPARK\_DRIVER\_PYTHON\_OPTS notebook  
> pyspark ----driver-class-path /path-to-jar-file...

Everything is fine. But I don't want to connect to Spark server very more. Maybe I can only:

> setx PYSPARK\_DRIVER\_PYTHON ipython  
> setx PYSPARK\_DRIVER\_PYTHON\_OPTS notebook  
> setx SPARK\_CLASSPATH / **path-to-all-jar-files...**  
> pyspark ----driver-class-path SPARK\_CLASSPATH

But all are not run. Error:

> Py4JJavaError: An error occurred while calling o139.save.  
> : java.lang.ClassNotFoundException: Failed to find data source: es. Please find packages at [Third-Party Projects | Apache Spark](http://spark.apache.org/third-party-projects.html)  
> at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:635)  
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:241)  
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:225)  
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
> at java.lang.reflect.Method.invoke(Method.java:498)  
> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)  
> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)  
> at py4j.Gateway.invoke(Gateway.java:282)  
> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)  
> at py4j.commands.CallCommand.execute(CallCommand.java:79)  
> at py4j.GatewayConnection.run(GatewayConnection.java:214)  
> at java.lang.Thread.run(Thread.java:748)  
> Caused by: java.lang.ClassNotFoundException: es.DefaultSource  
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)  
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)  
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)  
> at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$23$$anonfun$apply$15.apply(DataSource.scala:618)  
> at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$23$$anonfun$apply$15.apply(DataSource.scala:618)  
> at scala.util.Try$.apply(Try.scala:192)  
> at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$23.apply(DataSource.scala:618)  
> at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$23.apply(DataSource.scala:618)  
> at scala.util.Try.orElse(Try.scala:84)  
> at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:618)  
> ... 13 more

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [May 30, 2018, 7:32pm UTC](https://discuss.elastic.co/t/how-to-run-es-hadoop-in-jupyter-notebook-python-or-scala/131456/2 "2018-05-30T19:32:01Z")

</div>

I would make sure that ES-Hadoop is on all of your classpaths. Alternatively, you could try using the full length name for the connector (`org.elasticsearch.spark.sql`) and give it a shot? I've seen some cases where the short name was unusable

---

<div class="post-metadata">

**Author:** ![Malek\_Soltani](https://avatars.discourse-cdn.com/v4/letter/m/7c8e57/32.png) [@Malek\_Soltani](https://discuss.elastic.co/u/Malek_Soltani)\
**Post date:** [June 9, 2018, 12:01am UTC](https://discuss.elastic.co/t/how-to-run-es-hadoop-in-jupyter-notebook-python-or-scala/131456/3 "2018-06-09T00:01:06Z")

</div>

you can specify the path to the jar file while starting pyspark :

> pyspark --jars /Path\_to \_ESHadoop\_sparkJar/YourESHadoop\_sparkJar.jar

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 7, 2018, 12:01am UTC](https://discuss.elastic.co/t/how-to-run-es-hadoop-in-jupyter-notebook-python-or-scala/131456/4 "2018-07-07T00:01:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.