Hortonworks distribution and ES 5.2

Elastic Stack Elasticsearch
1

We currently load data to ES using a Spark script running on a Hortonworks cluster HDP 2.4 (Spark 1.6). We have a new dev ES cluster 5.2. I got this error when running the same script to load the data to ES 5.2 cluster:

org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Unsupported/Unknown Elasticsearch version 5.2.0

I checked the latest Hortonworks distro, and it is also using Spark 1.6.

Any way to use Hortonworks with ES 5.2?

2

@eman you will need to upgrade your version of ES-Hadoop to version 5.2.0. ES-Hadoop 5.2.0 is backwards compatible with earlier versions of Elasticsearch, so it should be fairly straightforward. Please also double check that there are no breaking changes that may cause you issues during the upgrade.

3

Thanks for the tip @james.baiera . I ended up using ES Spark connector for Spark 1.3 to 1.6
"org.elasticsearch" % "elasticsearch-spark-13_2.10" % "5.2.0"

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.