https://spark-packages.org/package/elastic/elasticsearch-hadoop that anything above 5.0.0-Alpha4 release provides only Scala 2.11 versions and does not have jars compiled for Scala 2.10 versions, Are there any specific dependancy on Scala 2.11 which forces this change? If not can we also generate Scala 2.10 version and upload them to Spark Packages so that it benefits users like us.
Currently we are in Spark 1.6.2, Scala 2.10 and ES 5.1.1. I am not able to use Elasticsearch Spark Connector. While we were trying out Alpha release, I was able to at least use 5.0.0.-Alpha4 connector which was with Scala 2.10. But I am not able to use them now given the fact that we do a check for version starting with 5.0 with InitializationUtils.java in org.elasticsearch.hadoop.rest in 5.0.0-Alpha4 which is preventing it to work from 5.1.1.
Please let me know what is the alternative
Also please confirm if latest release of connectors will also support Spark 1.6.2.
@krishmah In ES-Hadoop 5.x we are choosing to target Spark 2.0 on Scala 2.11 by default. We provide a set of legacy jars for users who are not on the later versions of Spark or Scala. I advise retrieving the jars from maven central that correspond to your Spark/Scala version(s).
I'm replying to this for future reference and for those that are having problems finding the correct jars/packages. I appreciate James' response but it's not complete.
Here you will find multiple directories for elasticsearch-spark. The directory elasticsearch-spark-13_* contains packages for spark version 1.3-1.6. Scala versions are appended to the end of the directory name.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.