Cannot read from Elasticsearch using Spark SQL


(michele crudele) #1

I wrote this simple notebook in scala using Elasticsearch Spark adapter:

%AddJar
file:///tools/elasticsearch-hadoop-2.1.0.Beta3/dist/elasticsearch-spark_2.10-2.1.0.BUILD-SNAPSHOT.jar
%AddJar
file:///tools/elasticsearch-hadoop-2.1.0.Beta3/dist/elasticsearch-hadoop-2.1.0.BUILD-SNAPSHOT.jar

// Write something to spark/docs index
//
import org.elasticsearch.spark._

val michelangelo = Map("artist" -> "Michelangelo", "bio" -> "Painter,
sculptor, architect born in Florence in... ")
val leonardo = Map("artist" -> "Leonardo", "bio" -> "Absolute genius;
painter, inventor born in the little village of Vinci...")

sc.makeRDD(Seq(michelangelo, leonardo)).saveToEs("spark/docs")

// Search for painters through spark/docs
//
val painters = sc.esRDD("spark/docs", "?q=painter")
println("Number of painters in spark/docs: " + painters.count())

painters.collect().foreach(x => println("ID: " + x._1))

// Try to read using SparkSQL
//
import org.apache.spark.sql.SQLContext
import org.elasticsearch.spark._
import org.elasticsearch.spark.sql._

val sql = new SQLContext(sc)

// Here is where I get an exception
//
val docsSql = sql.esRDD("spark/docs")

Name: java.lang.NoSuchMethodError
Message:
org.apache.spark.sql.catalyst.types.StructField.(Ljava/lang/String;Lorg/apache/spark/sql/catalyst/types/DataType;Z)V
StackTrace:
org.elasticsearch.spark.sql.MappingUtils$.org$elasticsearch$spark$sql$MappingUtils$$convertField(MappingUtils.scala:75)
org.elasticsearch.spark.sql.MappingUtils$$anonfun$convertToStruct$1.apply(MappingUtils.scala:54)
org.elasticsearch.spark.sql.MappingUtils$$anonfun$convertToStruct$1.apply(MappingUtils.scala:54)
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
org.elasticsearch.spark.sql.MappingUtils$.convertToStruct(MappingUtils.scala:54)
org.elasticsearch.spark.sql.MappingUtils$.discoverMapping(MappingUtils.scala:47)
org.elasticsearch.spark.sql.EsSparkSQL$.esRDD(EsSparkSQL.scala:27)
org.elasticsearch.spark.sql.EsSparkSQL$.esRDD(EsSparkSQL.scala:23)
org.elasticsearch.spark.sql.package$SQLContextFunctions.esRDD(package.scala:16)
$line460.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$$$$$c57ec8bf9b0d5f6161b97741d596ff0$$$$wC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:111)
$line460.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:116)
$line460.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:118)
$line460.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$

What am I doing wrong? Thanks

  • Michele

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/49a5c624-bda0-4835-98bc-915ec80d4fa3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Costin Leau) #2

Beta3 work with Spark SQL 1.0 and 1.1
Spark SQL 1.2 was released after that and broke binary backwards compatibility however his has been fixed in master/dev
version [1]
Note that Spark SQL 1.3 was released as well and again, broke backwards compatibility this time significant hence why
there are now two versions [2] - make sure to use the one appropriate for your Spark version.

Cheers,

[1] http://www.elastic.co/guide/en/elasticsearch/hadoop/master/install.html#download-dev
[2] http://www.elastic.co/guide/en/elasticsearch/hadoop/master/spark.html#spark-sql-versions

On 4/20/15 1:17 PM, michele crudele wrote:

I wrote this simple notebook in scala using Elasticsearch Spark adapter:

%AddJar file:///tools/elasticsearch-hadoop-2.1.0.Beta3/dist/elasticsearch-spark_2.10-2.1.0.BUILD-SNAPSHOT.jar
%AddJar file:///tools/elasticsearch-hadoop-2.1.0.Beta3/dist/elasticsearch-hadoop-2.1.0.BUILD-SNAPSHOT.jar

// Write something to spark/docs index
//
import org.elasticsearch.spark._

val michelangelo = Map("artist" -> "Michelangelo", "bio" -> "Painter, sculptor, architect born in Florence in... ")
val leonardo = Map("artist" -> "Leonardo", "bio" -> "Absolute genius; painter, inventor born in the little village of
Vinci...")

sc.makeRDD(Seq(michelangelo, leonardo)).saveToEs("spark/docs")

// Search for painters through spark/docs
//
val painters = sc.esRDD("spark/docs", "?q=painter")
println("Number of painters in spark/docs: " + painters.count())

painters.collect().foreach(x => println("ID: " + x._1))

// Try to read using SparkSQL
//
import org.apache.spark.sql.SQLContext
import org.elasticsearch.spark._
import org.elasticsearch.spark.sql._

val sql = new SQLContext(sc)

// Here is where I get an exception
//
val docsSql = sql.esRDD("spark/docs")

Name: java.lang.NoSuchMethodError
Message:
org.apache.spark.sql.catalyst.types.StructField.(Ljava/lang/String;Lorg/apache/spark/sql/catalyst/types/DataType;Z)V
StackTrace:
org.elasticsearch.spark.sql.MappingUtils$.org$elasticsearch$spark$sql$MappingUtils$$convertField(MappingUtils.scala:75)
org.elasticsearch.spark.sql.MappingUtils$$anonfun$convertToStruct$1.apply(MappingUtils.scala:54)
org.elasticsearch.spark.sql.MappingUtils$$anonfun$convertToStruct$1.apply(MappingUtils.scala:54)
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
org.elasticsearch.spark.sql.MappingUtils$.convertToStruct(MappingUtils.scala:54)
org.elasticsearch.spark.sql.MappingUtils$.discoverMapping(MappingUtils.scala:47)
org.elasticsearch.spark.sql.EsSparkSQL$.esRDD(EsSparkSQL.scala:27)
org.elasticsearch.spark.sql.EsSparkSQL$.esRDD(EsSparkSQL.scala:23)
org.elasticsearch.spark.sql.package$SQLContextFunctions.esRDD(package.scala:16)
$line460.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$$$$$c57ec8bf9b0d5f6161b97741d596ff0$$$$wC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:111)
$line460.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:116)
$line460.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:118)
$line460.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$

What am I doing wrong? Thanks

  • Michele

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/49a5c624-bda0-4835-98bc-915ec80d4fa3%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/49a5c624-bda0-4835-98bc-915ec80d4fa3%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5534D710.3020402%40gmail.com.
For more options, visit https://groups.google.com/d/optout.


(michele crudele) #3

Thanks Costin.

Il giorno lunedì 20 aprile 2015 12:38:30 UTC+2, Costin Leau ha scritto:

Beta3 work with Spark SQL 1.0 and 1.1
Spark SQL 1.2 was released after that and broke binary backwards
compatibility however his has been fixed in master/dev
version [1]
Note that Spark SQL 1.3 was released as well and again, broke backwards
compatibility this time significant hence why
there are now two versions [2] - make sure to use the one appropriate for
your Spark version.

Cheers,

[1]
http://www.elastic.co/guide/en/elasticsearch/hadoop/master/install.html#download-dev
[2]
http://www.elastic.co/guide/en/elasticsearch/hadoop/master/spark.html#spark-sql-versions

On 4/20/15 1:17 PM, michele crudele wrote:

I wrote this simple notebook in scala using Elasticsearch Spark adapter:

%AddJar
file:///tools/elasticsearch-hadoop-2.1.0.Beta3/dist/elasticsearch-spark_2.10-2.1.0.BUILD-SNAPSHOT.jar

%AddJar
file:///tools/elasticsearch-hadoop-2.1.0.Beta3/dist/elasticsearch-hadoop-2.1.0.BUILD-SNAPSHOT.jar

// Write something to spark/docs index
//
import org.elasticsearch.spark._

val michelangelo = Map("artist" -> "Michelangelo", "bio" -> "Painter,
sculptor, architect born in Florence in... ")
val leonardo = Map("artist" -> "Leonardo", "bio" -> "Absolute genius;
painter, inventor born in the little village of
Vinci...")

sc.makeRDD(Seq(michelangelo, leonardo)).saveToEs("spark/docs")

// Search for painters through spark/docs
//
val painters = sc.esRDD("spark/docs", "?q=painter")
println("Number of painters in spark/docs: " + painters.count())

painters.collect().foreach(x => println("ID: " + x._1))

// Try to read using SparkSQL
//
import org.apache.spark.sql.SQLContext
import org.elasticsearch.spark._
import org.elasticsearch.spark.sql._

val sql = new SQLContext(sc)

// Here is where I get an exception
//
val docsSql = sql.esRDD("spark/docs")

Name: java.lang.NoSuchMethodError
Message:

org.apache.spark.sql.catalyst.types.StructField.(Ljava/lang/String;Lorg/apache/spark/sql/catalyst/types/DataType;Z)V

StackTrace:

org.elasticsearch.spark.sql.MappingUtils$.org$elasticsearch$spark$sql$MappingUtils$$convertField(MappingUtils.scala:75)

org.elasticsearch.spark.sql.MappingUtils$$anonfun$convertToStruct$1.apply(MappingUtils.scala:54)

org.elasticsearch.spark.sql.MappingUtils$$anonfun$convertToStruct$1.apply(MappingUtils.scala:54)

scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)

scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)

scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)

scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)

org.elasticsearch.spark.sql.MappingUtils$.convertToStruct(MappingUtils.scala:54)

org.elasticsearch.spark.sql.MappingUtils$.discoverMapping(MappingUtils.scala:47)

org.elasticsearch.spark.sql.EsSparkSQL$.esRDD(EsSparkSQL.scala:27)
org.elasticsearch.spark.sql.EsSparkSQL$.esRDD(EsSparkSQL.scala:23)

org.elasticsearch.spark.sql.package$SQLContextFunctions.esRDD(package.scala:16)

$line460.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$$$$$c57ec8bf9b0d5f6161b97741d596ff0$$$$wC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:111)

$line460.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:116)

$line460.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:118)

$line460.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$

What am I doing wrong? Thanks

  • Michele

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to
elasticsearc...@googlegroups.com <javascript:> <mailto:
elasticsearch+unsubscribe@googlegroups.com <javascript:>>.
To view this discussion on the web visit

https://groups.google.com/d/msgid/elasticsearch/49a5c624-bda0-4835-98bc-915ec80d4fa3%40googlegroups.com

<
https://groups.google.com/d/msgid/elasticsearch/49a5c624-bda0-4835-98bc-915ec80d4fa3%40googlegroups.com?utm_medium=email&utm_source=footer>.

For more options, visit https://groups.google.com/d/optout.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fbbc4fc9-6662-4d57-b189-cbf8c60f36af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Thomas Decaux) #4

Please Justin, put a simple HTML table in documentation with compatibilities matrix between Hadoop / Spark / elasticsearch-hadoop / elasticsearch !


(system) #5