This looks similar to Elasticsearch-hadoop Scala version update to 2.12 · Issue #1949 · elastic/elasticsearch-hadoop · GitHub. We don't make it clear enough in our documentation yet, but the Elasticsearch-spark jar in the big Elasticsearch-hadoop artifact only works spark 2.x with scala 2.11. In that spark jar name (Elasticsearch-spark-20_2.11-7.17.3.jar), the first number (20) is code for spark version 2.x. The second number (2.11) is the scala version. The third number (7.17.3) is the Elasticsearch version. Unfortunately neither scala nor spark tend to be compatible across versions.
So for spark 3.2.1 you'll want elasticsearch-spark-30_2.12. You'll have to use scala 2.12 if you are on 7.17.3. Scala 2.13 was only added in es-hadoop 8.1.0.
1 Like