# ES + spark 1.0.1 - unable to send RDDs to ES **URL:** \ **Category:** Elasticsearch\ **Created:** [August 6, 2014, 4:32pm UTC](https://discuss.elastic.co/t/es-spark-1-0-1-unable-to-send-rdds-to-es/19120 "2014-08-06T16:32:01Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![philg](https://avatars.discourse-cdn.com/v4/letter/p/a88e57/32.png) [@philg](https://discuss.elastic.co/u/philg)\ **Post date:** [August 6, 2014, 4:32pm UTC](https://discuss.elastic.co/t/es-spark-1-0-1-unable-to-send-rdds-to-es/19120/1 "2014-08-06T16:32:01Z")
hello my context : spark, spark-shell 1.0.1 jdk1.7 scala 2.10.4, ES-Hadoop 2.1.0 ( nighly build) my problem: - unable to send RDDs from spark to ES i got a NoClassDefFoundError see below ( org/codehaus/jackson/annotate/JsonClass) jackson Jars to add to spark shell? philippe best regards * * * $bin/spark-shell --jars /usr/lib/spark-1.0/lib/elasticsearch-hadoop-2.1.0.jar ...... spark version 1.0.1 Using Scala version 2.10.4 .............. 14/08/06 17:19:36 INFO SparkContext: Added JAR file:/usr/lib/spark-1.0/lib/elasticsearch-hadoop-2.1.0.jar scala\> import org.elasticsearch.hadoop.mr.EsOutputFormat import org.elasticsearch.hadoop.mr.EsInputFormat import org.elasticsearch.hadoop.cfg.ConfigurationOptions import org.apache.hadoop.mapred.{FileOutputCommitter, FileOutputFormat, JobConf, OutputFormat} import org.apache.hadoop.fs.Path import org.apache.hadoop.io.{MapWritable, Text, NullWritable} val jobConf = new JobConf(sc.hadoopConfiguration) jobConf.set("es.resource", "myindex/mytype") jobConf.set("es.query", "?q=_") val esRDD = sc.hadoopRDD(jobConf,classOf[EsInputFormat[Text, MapWritable]],classOf[Text],classOf[MapWritable]) //_ up to there everything ok \* es.count() ----\> java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass at org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findDeserializationType(JacksonAnnotationIntrospector.java:524) ......... at org.elasticsearch.hadoop.rest.RestClient.parseContent(RestClient.java:119) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f525de7a-2d5f-41d0-b284-9ed2886c8c22%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f525de7a-2d5f-41d0-b284-9ed2886c8c22%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![philg](https://avatars.discourse-cdn.com/v4/letter/p/a88e57/32.png) [@philg](https://discuss.elastic.co/u/philg)\ **Post date:** [August 6, 2014, 4:40pm UTC](https://discuss.elastic.co/t/es-spark-1-0-1-unable-to-send-rdds-to-es/19120/2 "2014-08-06T16:40:22Z")
sorry for the mistake : --\> unable to read from ES and create RDDS On Wednesday, August 6, 2014 6:32:02 PM UTC+2, Phil gib wrote: > hello > my context : spark, spark-shell 1.0.1 jdk1.7 scala 2.10.4, ES-Hadoop > 2.1.0 ( nighly build) > my problem: > > - unable to read from ES and create RDDS > i got a NoClassDefFoundError see below ( > org/codehaus/jackson/annotate/JsonClass) > jackson Jars to add to spark shell? > > philippe > best regards -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/63db7ae7-0da9-4b87-a835-42e38f11ee05%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/63db7ae7-0da9-4b87-a835-42e38f11ee05%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [August 6, 2014, 4:46pm UTC](https://discuss.elastic.co/t/es-spark-1-0-1-unable-to-send-rdds-to-es/19120/3 "2014-08-06T16:46:40Z")
Hi Phil, Glad to see the work in es-hadoop master is being picked up even without any public announcement of it 馃檪 The issue has been fixed in master [1] and already pushed to Maven - can you please update and try again? FTR: The issue seems to be caused by multiple versions of Jackson which are pulled in the classpath (one from Hadoop, another from Spark) which on some platforms, causes class loading issues in Jackson during start-up. The fix in master hopefully remedies that. Cheers, [1] [jackson-core-asl version 路 Issue #239 路 elastic/elasticsearch-hadoop 路 GitHub](https://github.com/elasticsearch/elasticsearch-hadoop/issues/239) On 8/6/14 7:32 PM, Phil gib wrote: > hello > my context : spark, spark-shell 1.0.1 jdk1.7 scala 2.10.4, ES-Hadoop 2.1.0 ( nighly build) > my problem: > > - unable to send RDDs from spark to ES > i got a NoClassDefFoundError see below ( org/codehaus/jackson/annotate/JsonClass) > jackson Jars to add to spark shell? > > philippe > best regards > > * * * > > $bin/spark-shell --jars /usr/lib/spark-1.0/lib/elasticsearch-hadoop-2.1.0.jar > ...... > spark version 1.0.1 > Using Scala version 2.10.4 > .............. > 14/08/06 17:19:36 INFO SparkContext: Added JAR file:/usr/lib/spark-1.0/lib/elasticsearch-hadoop-2.1.0.jar > scala\> > import org.elasticsearch.hadoop.mr.EsOutputFormat > import org.elasticsearch.hadoop.mr.EsInputFormat > import org.elasticsearch.hadoop.cfg.ConfigurationOptions > import org.apache.hadoop.mapred.{FileOutputCommitter, FileOutputFormat, JobConf, OutputFormat} > import org.apache.hadoop.fs.Path > import [org.apache.hadoop.io](http://org.apache.hadoop.io).{MapWritable, Text, NullWritable} > val jobConf = new JobConf(sc.hadoopConfiguration) jobConf.set("es.resource", "myindex/mytype") > jobConf.set("es.query", "?q=\*") > val esRDD = sc.hadoopRDD(jobConf,classOf[EsInputFormat[Text, MapWritable]],classOf[Text],classOf[MapWritable]) > //\*up to there everything ok \* > es.count() > ----\> > java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass > at > org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findDeserializationType(JacksonAnnotationIntrospector.java:524) > ......... > at org.elasticsearch.hadoop.rest.RestClient.parseContent(RestClient.java:119) > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/f525de7a-2d5f-41d0-b284-9ed2886c8c22%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f525de7a-2d5f-41d0-b284-9ed2886c8c22%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/f525de7a-2d5f-41d0-b284-9ed2886c8c22%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f525de7a-2d5f-41d0-b284-9ed2886c8c22%40googlegroups.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/53E25BF0.4000307%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/53E25BF0.4000307%40gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![philg](https://avatars.discourse-cdn.com/v4/letter/p/a88e57/32.png) [@philg](https://discuss.elastic.co/u/philg)\ **Post date:** [August 7, 2014, 7:59am UTC](https://discuss.elastic.co/t/es-spark-1-0-1-unable-to-send-rdds-to-es/19120/4 "2014-08-07T07:59:11Z")
Hi Costin Thanks for your help! it works now after the update 馃檪 Best Regards phil On Wednesday, August 6, 2014 6:46:57 PM UTC+2, Costin Leau wrote: > Hi Phil, > > Glad to see the work in es-hadoop master is being picked up even without > any public announcement of it 馃檪 > > The issue has been fixed in master [1] and already pushed to Maven - can > you please update and try again? > > FTR: The issue seems to be caused by multiple versions of Jackson which > are pulled in the classpath (one from Hadoop, > another from Spark) > which on some platforms, causes class loading issues in Jackson during > start-up. The fix in master hopefully remedies that. > > Cheers, > > [1] [jackson-core-asl version 路 Issue #239 路 elastic/elasticsearch-hadoop 路 GitHub](https://github.com/elasticsearch/elasticsearch-hadoop/issues/239) > > On 8/6/14 7:32 PM, Phil gib wrote: > > > hello > > my context : spark, spark-shell 1.0.1 jdk1.7 scala 2.10.4, ES-Hadoop > > 2.1.0 ( nighly build) > > my problem: > > > > - unable to send RDDs from spark to ES > > i got a NoClassDefFoundError see below ( > > org/codehaus/jackson/annotate/JsonClass) > > jackson Jars to add to spark shell? > > > > philippe > > best regards > > > > * * * > > > > $bin/spark-shell --jars > > /usr/lib/spark-1.0/lib/elasticsearch-hadoop-2.1.0.jar > > ...... > > spark version 1.0.1 > > Using Scala version 2.10.4 > > .............. > > 14/08/06 17:19:36 INFO SparkContext: Added JAR > > file:/usr/lib/spark-1.0/lib/elasticsearch-hadoop-2.1.0.jar > > scala\> > > import org.elasticsearch.hadoop.mr.EsOutputFormat > > import org.elasticsearch.hadoop.mr.EsInputFormat > > import org.elasticsearch.hadoop.cfg.ConfigurationOptions > > import org.apache.hadoop.mapred.{FileOutputCommitter, FileOutputFormat, > > JobConf, OutputFormat} > > import org.apache.hadoop.fs.Path > > import [org.apache.hadoop.io](http://org.apache.hadoop.io).{MapWritable, Text, NullWritable} > > val jobConf = new JobConf(sc.hadoopConfiguration) > > jobConf.set("es.resource", "myindex/mytype") > > jobConf.set("es.query", "?q=\*") > > val esRDD = sc.hadoopRDD(jobConf,classOf[EsInputFormat[Text, > > MapWritable]],classOf[Text],classOf[MapWritable]) > > //\*up to there everything ok \* > > es.count() > > ----\> > > java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass > > at > > org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findDeserializationType(JacksonAnnotationIntrospector.java:524) > > > ``` > > ......... > > at > > > > ``` > > org.elasticsearch.hadoop.rest.RestClient.parseContent(RestClient.java:119) > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \ \ > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \\>. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/f525de7a-2d5f-41d0-b284-9ed2886c8c22%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f525de7a-2d5f-41d0-b284-9ed2886c8c22%40googlegroups.com) > > > \< > > [https://groups.google.com/d/msgid/elasticsearch/f525de7a-2d5f-41d0-b284-9ed2886c8c22%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f525de7a-2d5f-41d0-b284-9ed2886c8c22%40googlegroups.com?utm_medium=email&utm_source=footer)\>. > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2ef2ba16-5a25-4b73-9f09-018e78f517e2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/2ef2ba16-5a25-4b73-9f09-018e78f517e2%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:10am UTC](https://discuss.elastic.co/t/es-spark-1-0-1-unable-to-send-rdds-to-es/19120/5 "2017-07-06T01:10:16Z")