Spark version, elasticsearch-hadoop version, akka version sync up

Elastic Stack Elasticsearch
1

There are plenty of spark / akka / scala / elasticsearch-hadoop
dependencies to keep track of.

Is it true that elasticsearch-hadoop needs to be compiled for a specific
spark version to run correctly on the cluster? I'm also trying to keep
track of the akka version and scala version. i.e, wil es-hadoop compiled
for spark 1.2 work with Spark 1.3 ?

When the elasticsearch-hadoop versions are released, as v2.0 v2.1,
v2.1.0.Beta3, at what point do we need to keep in mind what spark version
it was also compiled against?
i.e., is it safe to assume the es-hadoop versions are tied to a specific
spark core version?

I've been keeping the following chart in my notes to see what all the
versions are with all dependencies

Akka Version Dependencies
Current Akka Stable Release: 2.3.9

Elasticsearch-Hadoop: 2.1.0Beta3 = Spark 1.1.0
Elasticsearch-Hadoop: 2.1.0Beta3-SNAPSHOT = Spark 1.2.1
Elasticsearch-Hadoop: what about spark 1.3 ?

Spark: 1.3, Akka: 2.3.4-spark
Spark: 1.2, Akka: 2.3.4-spark
Spark: 1.1, Akka: 2.2.3-shaded-protobuf

Activator 1.2.12 comes with with Akka 2.3.4

Play 2.3.8, akka 2.3.4, scala 2.11.1 (will also work with 2.10.4 )
Play 2.2.x, akka 2.2.0

Spark Job Server 0.4.1, Spark Core 1.1.0, Akka, 2.2.4
Spark Job Server Master as of Feb 22, 2015, Spark Core 1.2.0, Akka 2.3.4,
Scala 2.10.4

Akka persistence latest 2.3.4 or later
Akka 2.3.9 is released for Scala 2.10.4 and 2.11.5

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/28ad3f78-8b3d-450a-a29d-06d3e6636cfd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

es-hadoop doesn't depend on akka, only on Spark. The scala version that
es-hadoop is compiled against matches the one used by the Spark version
compiled against for each release - typically this shouldn't pose a problem.

Unfortunately, despite the minor version increments, some of the Spark APIs
or components (in particular Spark SQL) have changed drastically between
each release breaking backwards compatibility. For example, Beta3 works
until Spark 1.1 (which was the latest stable release during its release)
but not with 1.2. This is fixed in master however the current dev build
doesn't work with Spark SQL in the newly released 1.3 (does work with Spark
core).

This has already been fixed locally however I'm having difficulties trying
to preserve compatibility across the Spark SQL 1.2 release and 1.3.

Long story short, as long as the dependencies for Spark are in order, the
same should apply for es-hadoop as well since it relies only on Spark (and
Scala of course).

On Tue, Mar 17, 2015 at 10:43 PM, Jeff Steinmetz <
jeffrey.steinmetz@gmail.com> wrote:

There are plenty of spark / akka / scala / elasticsearch-hadoop
dependencies to keep track of.

Is it true that elasticsearch-hadoop needs to be compiled for a specific
spark version to run correctly on the cluster? I'm also trying to keep
track of the akka version and scala version. i.e, wil es-hadoop compiled
for spark 1.2 work with Spark 1.3 ?

When the elasticsearch-hadoop versions are released, as v2.0 v2.1,
v2.1.0.Beta3, at what point do we need to keep in mind what spark version
it was also compiled against?
i.e., is it safe to assume the es-hadoop versions are tied to a specific
spark core version?

I've been keeping the following chart in my notes to see what all the
versions are with all dependencies

Akka Version Dependencies
Current Akka Stable Release: 2.3.9

Elasticsearch-Hadoop: 2.1.0Beta3 = Spark 1.1.0
Elasticsearch-Hadoop: 2.1.0Beta3-SNAPSHOT = Spark 1.2.1
Elasticsearch-Hadoop: what about spark 1.3 ?

Spark: 1.3, Akka: 2.3.4-spark
Spark: 1.2, Akka: 2.3.4-spark
Spark: 1.1, Akka: 2.2.3-shaded-protobuf

Activator 1.2.12 comes with with Akka 2.3.4

Play 2.3.8, akka 2.3.4, scala 2.11.1 (will also work with 2.10.4 )
Play 2.2.x, akka 2.2.0

Spark Job Server 0.4.1, Spark Core 1.1.0, Akka, 2.2.4
Spark Job Server Master as of Feb 22, 2015, Spark Core 1.2.0, Akka 2.3.4,
Scala 2.10.4

Akka persistence latest 2.3.4 or later
Akka 2.3.9 is released for Scala 2.10.4 and 2.11.5

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/28ad3f78-8b3d-450a-a29d-06d3e6636cfd%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/28ad3f78-8b3d-450a-a29d-06d3e6636cfd%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJogdmf9z8JjP_LWTjObDuKiE4DzdSOH2rA%3DdbtkfqQakXYbkw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Thank you for the summary - you are confirming (as a sanity check for
myself):

elasticsearch-hadoop beta3 (not snapshot) on spark core 1.1 only
elasticsearch-hadoop-beta3-SNAPSHOT with spark core 1.1, 1.2 and 1.3 -- as
long as I don't use Spark SQL when using 1.2 and 1.3

Costin - I am amazed by your ability to keep all this straight - my head
would explode dealing with all the dependencies in flux. Kudos to you.

On Tuesday, March 17, 2015 at 2:12:06 PM UTC-7, Costin Leau wrote:

es-hadoop doesn't depend on akka, only on Spark. The scala version that
es-hadoop is compiled against matches the one used by the Spark version
compiled against for each release - typically this shouldn't pose a problem.

Unfortunately, despite the minor version increments, some of the Spark
APIs or components (in particular Spark SQL) have changed drastically
between each release breaking backwards compatibility. For example, Beta3
works until Spark 1.1 (which was the latest stable release during its
release) but not with 1.2. This is fixed in master however the current dev
build doesn't work with Spark SQL in the newly released 1.3 (does work with
Spark core).

This has already been fixed locally however I'm having difficulties trying
to preserve compatibility across the Spark SQL 1.2 release and 1.3.

Long story short, as long as the dependencies for Spark are in order, the
same should apply for es-hadoop as well since it relies only on Spark (and
Scala of course).

On Tue, Mar 17, 2015 at 10:43 PM, Jeff Steinmetz <jeffrey....@gmail.com
<javascript:>> wrote:

There are plenty of spark / akka / scala / elasticsearch-hadoop
dependencies to keep track of.

Is it true that elasticsearch-hadoop needs to be compiled for a specific
spark version to run correctly on the cluster? I'm also trying to keep
track of the akka version and scala version. i.e, wil es-hadoop compiled
for spark 1.2 work with Spark 1.3 ?

When the elasticsearch-hadoop versions are released, as v2.0 v2.1,
v2.1.0.Beta3, at what point do we need to keep in mind what spark version
it was also compiled against?
i.e., is it safe to assume the es-hadoop versions are tied to a specific
spark core version?

I've been keeping the following chart in my notes to see what all the
versions are with all dependencies

Akka Version Dependencies
Current Akka Stable Release: 2.3.9

Elasticsearch-Hadoop: 2.1.0Beta3 = Spark 1.1.0
Elasticsearch-Hadoop: 2.1.0Beta3-SNAPSHOT = Spark 1.2.1
Elasticsearch-Hadoop: what about spark 1.3 ?

Spark: 1.3, Akka: 2.3.4-spark
Spark: 1.2, Akka: 2.3.4-spark
Spark: 1.1, Akka: 2.2.3-shaded-protobuf

Activator 1.2.12 comes with with Akka 2.3.4

Play 2.3.8, akka 2.3.4, scala 2.11.1 (will also work with 2.10.4 )
Play 2.2.x, akka 2.2.0

Spark Job Server 0.4.1, Spark Core 1.1.0, Akka, 2.2.4
Spark Job Server Master as of Feb 22, 2015, Spark Core 1.2.0, Akka
2.3.4, Scala 2.10.4

Akka persistence latest 2.3.4 or later
Akka 2.3.9 is released for Scala 2.10.4 and 2.11.5

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/28ad3f78-8b3d-450a-a29d-06d3e6636cfd%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/28ad3f78-8b3d-450a-a29d-06d3e6636cfd%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0c19d1fa-17b9-4e6e-a698-b49c7d6919d3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

You're close:
elasticsearch-hadoop snapshot (aka dev aka master) works on spark 1.2, 1.1
and 1.0, both core and sql
elasticsearch-hadoop beta3 (not snapshot) works on spark 1.1 and spark 1.0,
both core and sql
elasticsearch-hadoop beta2 (not snapshot) works on spark 1.0 (core and sql)

The support for spark 1.3 hasn't been committed yet, I'd like to push it
out once I (hopefully) manage to keep compatibility on the sql integration
with spark 1.2 and lower. Once it is, it will be available in the
nightly/dev builds published (as the name implies) every 24th - each day or
night, depending on your timezon [1].

Costin - I am amazed by your ability to keep all this straight - my head
would explode dealing with all the dependencies in flux. Kudos to you.

Thanks. As most (if not all) things in life, practice makes perfect :wink:

[1]

On Wed, Mar 18, 2015 at 12:59 AM, Jeff Steinmetz <
jeffrey.steinmetz@gmail.com> wrote:

Thank you for the summary - you are confirming (as a sanity check for
myself):

elasticsearch-hadoop beta3 (not snapshot) on spark core 1.1 only
elasticsearch-hadoop-beta3-SNAPSHOT with spark core 1.1, 1.2 and 1.3 -- as
long as I don't use Spark SQL when using 1.2 and 1.3

Costin - I am amazed by your ability to keep all this straight - my head
would explode dealing with all the dependencies in flux. Kudos to you.

On Tuesday, March 17, 2015 at 2:12:06 PM UTC-7, Costin Leau wrote:

es-hadoop doesn't depend on akka, only on Spark. The scala version that
es-hadoop is compiled against matches the one used by the Spark version
compiled against for each release - typically this shouldn't pose a problem.

Unfortunately, despite the minor version increments, some of the Spark
APIs or components (in particular Spark SQL) have changed drastically
between each release breaking backwards compatibility. For example, Beta3
works until Spark 1.1 (which was the latest stable release during its
release) but not with 1.2. This is fixed in master however the current dev
build doesn't work with Spark SQL in the newly released 1.3 (does work with
Spark core).

This has already been fixed locally however I'm having difficulties
trying to preserve compatibility across the Spark SQL 1.2 release and 1.3.

Long story short, as long as the dependencies for Spark are in order, the
same should apply for es-hadoop as well since it relies only on Spark (and
Scala of course).

On Tue, Mar 17, 2015 at 10:43 PM, Jeff Steinmetz jeffrey....@gmail.com
wrote:

There are plenty of spark / akka / scala / elasticsearch-hadoop
dependencies to keep track of.

Is it true that elasticsearch-hadoop needs to be compiled for a specific
spark version to run correctly on the cluster? I'm also trying to keep
track of the akka version and scala version. i.e, wil es-hadoop compiled
for spark 1.2 work with Spark 1.3 ?

When the elasticsearch-hadoop versions are released, as v2.0 v2.1,
v2.1.0.Beta3, at what point do we need to keep in mind what spark version
it was also compiled against?
i.e., is it safe to assume the es-hadoop versions are tied to a specific
spark core version?

I've been keeping the following chart in my notes to see what all the
versions are with all dependencies

Akka Version Dependencies
Current Akka Stable Release: 2.3.9

Elasticsearch-Hadoop: 2.1.0Beta3 = Spark 1.1.0
Elasticsearch-Hadoop: 2.1.0Beta3-SNAPSHOT = Spark 1.2.1
Elasticsearch-Hadoop: what about spark 1.3 ?

Spark: 1.3, Akka: 2.3.4-spark
Spark: 1.2, Akka: 2.3.4-spark
Spark: 1.1, Akka: 2.2.3-shaded-protobuf

Activator 1.2.12 comes with with Akka 2.3.4

Play 2.3.8, akka 2.3.4, scala 2.11.1 (will also work with 2.10.4 )
Play 2.2.x, akka 2.2.0

Spark Job Server 0.4.1, Spark Core 1.1.0, Akka, 2.2.4
Spark Job Server Master as of Feb 22, 2015, Spark Core 1.2.0, Akka
2.3.4, Scala 2.10.4

Akka persistence latest 2.3.4 or later
Akka 2.3.9 is released for Scala 2.10.4 and 2.11.5

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/28ad3f78-8b3d-450a-a29d-06d3e6636cfd%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/28ad3f78-8b3d-450a-a29d-06d3e6636cfd%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0c19d1fa-17b9-4e6e-a698-b49c7d6919d3%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/0c19d1fa-17b9-4e6e-a698-b49c7d6919d3%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJogdmf9-Fs4xDH9wVoVxvOG3CMSZ_XfCZb7%2BLZDT1AvDnxrZg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.