Elasticsearch-hadoop Spark 4 compatible release timelines

Elastic Stack Elasticsearch
1

Hi,

I'd like to know when a Spark 4 compatible release of GitHub - elastic/elasticsearch-hadoop: Elasticsearch real-time search and analytics natively integrated with Hadoop · GitHub is expected? I believe this might be 9.6?

Thanks in advance.

Nathan

2

Hi can i know the importance to it

3

Hi Etiosa,

We're in the process of upgrading from Spark 3 to Spark 4, and this is one of the few libraries we use which isn't yet publishing Spark 4 versions. If there's a version being released soon (it looks like work has very much started landing into main) then I won't look at replacing it in our stack; if there isn't then I'll have to look at options.

Nathan

4

Yes, support for Spark 3.5, 4.0, 4.1, and 4.2 will be in 9.6.0.

5

Super, and is there a target release date for that?

6

Unfortunately, Elastic does not publish target release dates. But you can probably get a good idea of when it will be based on the cadence of past minor releases.

7

Fair enough, looks like it might be a little while.

Do you publish snapshot/nightlies to anywhere publically accessible?

8

I don't know whether we publish snapshot artifacts anywhere. If you're comfortable building from source, you can clone the repository at GitHub - elastic/elasticsearch-hadoop: Elasticsearch real-time search and analytics natively integrated with Hadoop · GitHub, and then build the spark jar you're interested in with something like ./gradlew :elasticsearch-spark-41:assemble (that 41 is for spark 4.1 support -- change it if you're using a different version of spark). Then you'll find the artifact at ./spark/sql-41/build/libs/elasticsearch-spark-41_2.13-9.6.0-SNAPSHOT.jar.