Problem between Spark and Elasticsearch

(Vincent Vost) #1

Hi :slight_smile:

I would like to use Spark with Elasticsearch, but I have some trouble for put everything together

I'm running 2 vm.
One with elasticsearch and the second with spark.

import org.apache.spark.SparkConf;

import java.util.Map;

public class MySparkElas {

public static void main(String[] args) {
    SparkConf conf = new SparkConf().setAppName("MySparkElas").setMaster("local");
    conf.set("", "true");
    conf.set("es.nodes", "");
    conf.set("es.port", "9200");
    JavaSparkContext sc = new JavaSparkContext(conf);
    JavaPairRDD<String, Map<String, Object>> esRDD = JavaEsSpark.esRDD(sc,"test/);
    System.out.print("Number of elements :  "+esRDD.count());


And I get this error :

16/06/06 12:14:03 INFO BlockManagerMasterEndpoint: Registering block manager localhost:37102 with 517.4 MB RAM, BlockManagerId(driver, localhost, 37102)
16/06/06 12:14:03 INFO BlockManagerMaster: Registered BlockManager
Exception in thread "main" java.lang.NoClassDefFoundError: org/elasticsearch/spark/rdd/api/java/JavaEsSpark
at com.seigneurin.spark.MySparkElas.main(
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
at java.lang.reflect.Method.invoke(
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException:
at java.lang.ClassLoader.loadClass(
at java.lang.ClassLoader.loadClass(
... 10 more

Do you have an idea why it didn't work ?
Thank you :slight_smile:

(Vincent Vost) #2

Ok! I found the problem.

I didn't take the dependencies with the JAR. Now I have an other error, but this one is okay :slight_smile:

(system) #3