Problem between Spark and Elasticsearch

Elastic Stack Elasticsearch
1

Hi :slight_smile:

I would like to use Spark with Elasticsearch, but I have some trouble for put everything together

I'm running 2 vm.
One with elasticsearch and the second with spark.

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.elasticsearch.spark.rdd.api.java.JavaEsSpark;

import java.util.Map;

public class MySparkElas {

public static void main(String[] args) {

    SparkConf conf = new SparkConf().setAppName("MySparkElas").setMaster("local");
    conf.set("es.index.auto.create", "true");
    conf.set("es.nodes", "192.168.1.14");
    conf.set("es.port", "9200");
    JavaSparkContext sc = new JavaSparkContext(conf);

    JavaPairRDD<String, Map<String, Object>> esRDD = JavaEsSpark.esRDD(sc,"test/);

    System.out.print("Number of elements :  "+esRDD.count());
}

}

And I get this error :

16/06/06 12:14:03 INFO BlockManagerMasterEndpoint: Registering block manager localhost:37102 with 517.4 MB RAM, BlockManagerId(driver, localhost, 37102)
16/06/06 12:14:03 INFO BlockManagerMaster: Registered BlockManager
Exception in thread "main" java.lang.NoClassDefFoundError: org/elasticsearch/spark/rdd/api/java/JavaEsSpark
at com.seigneurin.spark.MySparkElas.main(MySparkElas.java:25)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.spark.rdd.api.java.JavaEsSpark
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 10 more

Do you have an idea why it didn't work ?
Thank you :slight_smile:

2

Ok! I found the problem.

I didn't take the dependencies with the JAR. Now I have an other error, but this one is okay :slight_smile: