Elastic Search Spark Connector for Spark 1.6, ES 5.1.1 and Scala 2.10


(Mahadevan Krishnan) #1

Hi Team,

Can you please confirm the following 2 questions

  1. see that from Spark Packages link

https://spark-packages.org/package/elastic/elasticsearch-hadoop that anything above 5.0.0-Alpha4 release provides only Scala 2.11 versions and does not have jars compiled for Scala 2.10 versions, Are there any specific dependancy on Scala 2.11 which forces this change? If not can we also generate Scala 2.10 version and upload them to Spark Packages so that it benefits users like us.

Currently we are in Spark 1.6.2, Scala 2.10 and ES 5.1.1. I am not able to use Elasticsearch Spark Connector. While we were trying out Alpha release, I was able to at least use 5.0.0.-Alpha4 connector which was with Scala 2.10. But I am not able to use them now given the fact that we do a check for version starting with 5.0 with InitializationUtils.java in org.elasticsearch.hadoop.rest in 5.0.0-Alpha4 which is preventing it to work from 5.1.1.
Please let me know what is the alternative

  1. Also please confirm if latest release of connectors will also support Spark 1.6.2.

Regards,
Mahadevan


(James Baiera) #2

@krishmah In ES-Hadoop 5.x we are choosing to target Spark 2.0 on Scala 2.11 by default. We provide a set of legacy jars for users who are not on the later versions of Spark or Scala. I advise retrieving the jars from maven central that correspond to your Spark/Scala version(s).


#3

I'm replying to this for future reference and for those that are having problems finding the correct jars/packages. I appreciate James' response but it's not complete.

Don't go here: https://spark-packages.org/package/elastic/elasticsearch-hadoop
Go here instead: http://repo1.maven.org/maven2/org/elasticsearch/

Here you will find multiple directories for elasticsearch-spark. The directory elasticsearch-spark-13_* contains packages for spark version 1.3-1.6. Scala versions are appended to the end of the directory name.

elasticsearch-spark-1.2/
elasticsearch-spark-1.2_2.10/
elasticsearch-spark-1.2_2.11/
elasticsearch-spark-13_2.10/
elasticsearch-spark-13_2.11/
elasticsearch-spark-20_2.10/
elasticsearch-spark-20_2.11/
elasticsearch-spark_2.10/
elasticsearch-spark_2.11/

The answer to the original post for which package to use for Spark 1.6, Scala 2.10, and ES 5.1.1 is...

http://repo1.maven.org/maven2/org/elasticsearch/elasticsearch-spark-13_2.10/5.1.1/elasticsearch-spark-13_2.10-5.1.1.jar

OR

--packages org.elasticsearch:elasticsearch-spark-13_2.10:5.1.1


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.