# Spark version, elasticsearch-hadoop version, akka version sync up **URL:** \ **Category:** Elasticsearch\ **Created:** [March 17, 2015, 8:43pm UTC](https://discuss.elastic.co/t/spark-version-elasticsearch-hadoop-version-akka-version-sync-up/22722 "2015-03-17T20:43:44Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Jeff\_Steinmetz](https://avatars.discourse-cdn.com/v4/letter/j/ecd19e/32.png) [@Jeff\_Steinmetz](https://discuss.elastic.co/u/Jeff_Steinmetz)\ **Post date:** [March 17, 2015, 8:43pm UTC](https://discuss.elastic.co/t/spark-version-elasticsearch-hadoop-version-akka-version-sync-up/22722/1 "2015-03-17T20:43:44Z")
There are plenty of spark / akka / scala / elasticsearch-hadoop dependencies to keep track of. Is it true that elasticsearch-hadoop needs to be compiled for a specific spark version to run correctly on the cluster? I'm also trying to keep track of the akka version and scala version. i.e, wil es-hadoop compiled for spark 1.2 work with Spark 1.3 ? When the elasticsearch-hadoop versions are released, as v2.0 v2.1, v2.1.0.Beta3, at what point do we need to keep in mind what spark version it was also compiled against? i.e., is it safe to assume the es-hadoop versions are tied to a specific spark core version? # I've been keeping the following chart in my notes to see what all the versions are with all dependencies Akka Version Dependencies Current Akka Stable Release: 2.3.9 Elasticsearch-Hadoop: 2.1.0Beta3 = Spark 1.1.0 Elasticsearch-Hadoop: 2.1.0Beta3-SNAPSHOT = Spark 1.2.1 Elasticsearch-Hadoop: what about spark 1.3 ? Spark: 1.3, Akka: 2.3.4-spark Spark: 1.2, Akka: 2.3.4-spark Spark: 1.1, Akka: 2.2.3-shaded-protobuf Activator 1.2.12 comes with with Akka 2.3.4 Play 2.3.8, akka 2.3.4, scala 2.11.1 (will also work with 2.10.4 ) Play 2.2.x, akka 2.2.0 Spark Job Server 0.4.1, Spark Core 1.1.0, Akka, 2.2.4 Spark Job Server Master as of Feb 22, 2015, Spark Core 1.2.0, Akka 2.3.4, Scala 2.10.4 Akka persistence latest 2.3.4 or later Akka 2.3.9 is released for Scala 2.10.4 and 2.11.5 -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/28ad3f78-8b3d-450a-a29d-06d3e6636cfd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/28ad3f78-8b3d-450a-a29d-06d3e6636cfd%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [March 17, 2015, 9:11pm UTC](https://discuss.elastic.co/t/spark-version-elasticsearch-hadoop-version-akka-version-sync-up/22722/2 "2015-03-17T21:11:57Z")
es-hadoop doesn't depend on akka, only on Spark. The scala version that es-hadoop is compiled against matches the one used by the Spark version compiled against for each release - typically this shouldn't pose a problem. Unfortunately, despite the minor version increments, some of the Spark APIs or components (in particular Spark SQL) have changed drastically between each release breaking backwards compatibility. For example, Beta3 works until Spark 1.1 (which was the latest stable release during its release) but not with 1.2. This is fixed in master however the current dev build doesn't work with Spark SQL in the newly released 1.3 (does work with Spark core). This has already been fixed locally however I'm having difficulties trying to preserve compatibility across the Spark SQL 1.2 release and 1.3. Long story short, as long as the dependencies for Spark are in order, the same should apply for es-hadoop as well since it relies only on Spark (and Scala of course). On Tue, Mar 17, 2015 at 10:43 PM, Jeff Steinmetz \< [jeffrey.steinmetz@gmail.com](mailto:jeffrey.steinmetz@gmail.com)\> wrote: > There are plenty of spark / akka / scala / elasticsearch-hadoop > dependencies to keep track of. > > Is it true that elasticsearch-hadoop needs to be compiled for a specific > spark version to run correctly on the cluster? I'm also trying to keep > track of the akka version and scala version. i.e, wil es-hadoop compiled > for spark 1.2 work with Spark 1.3 ? > > When the elasticsearch-hadoop versions are released, as v2.0 v2.1, > v2.1.0.Beta3, at what point do we need to keep in mind what spark version > it was also compiled against? > i.e., is it safe to assume the es-hadoop versions are tied to a specific > spark core version? > > # I've been keeping the following chart in my notes to see what all the versions are with all dependencies > > Akka Version Dependencies > Current Akka Stable Release: 2.3.9 > > Elasticsearch-Hadoop: 2.1.0Beta3 = Spark 1.1.0 > Elasticsearch-Hadoop: 2.1.0Beta3-SNAPSHOT = Spark 1.2.1 > Elasticsearch-Hadoop: what about spark 1.3 ? > > Spark: 1.3, Akka: 2.3.4-spark > Spark: 1.2, Akka: 2.3.4-spark > Spark: 1.1, Akka: 2.2.3-shaded-protobuf > > Activator 1.2.12 comes with with Akka 2.3.4 > > Play 2.3.8, akka 2.3.4, scala 2.11.1 (will also work with 2.10.4 ) > Play 2.2.x, akka 2.2.0 > > Spark Job Server 0.4.1, Spark Core 1.1.0, Akka, 2.2.4 > Spark Job Server Master as of Feb 22, 2015, Spark Core 1.2.0, Akka 2.3.4, > Scala 2.10.4 > > Akka persistence latest 2.3.4 or later > Akka 2.3.9 is released for Scala 2.10.4 and 2.11.5 > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/28ad3f78-8b3d-450a-a29d-06d3e6636cfd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/28ad3f78-8b3d-450a-a29d-06d3e6636cfd%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/28ad3f78-8b3d-450a-a29d-06d3e6636cfd%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/28ad3f78-8b3d-450a-a29d-06d3e6636cfd%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJogdmf9z8JjP\_LWTjObDuKiE4DzdSOH2rA%3DdbtkfqQakXYbkw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJogdmf9z8JjP_LWTjObDuKiE4DzdSOH2rA%3DdbtkfqQakXYbkw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Jeff\_Steinmetz](https://avatars.discourse-cdn.com/v4/letter/j/ecd19e/32.png) [@Jeff\_Steinmetz](https://discuss.elastic.co/u/Jeff_Steinmetz)\ **Post date:** [March 17, 2015, 10:59pm UTC](https://discuss.elastic.co/t/spark-version-elasticsearch-hadoop-version-akka-version-sync-up/22722/3 "2015-03-17T22:59:30Z")
Thank you for the summary - you are confirming (as a sanity check for myself): elasticsearch-hadoop beta3 (not snapshot) on spark core 1.1 only elasticsearch-hadoop-beta3-SNAPSHOT with spark core 1.1, 1.2 and 1.3 -- as long as I don't use Spark SQL when using 1.2 and 1.3 Costin - I am amazed by your ability to keep all this straight - my head would explode dealing with all the dependencies in flux. Kudos to you. On Tuesday, March 17, 2015 at 2:12:06 PM UTC-7, Costin Leau wrote: > es-hadoop doesn't depend on akka, only on Spark. The scala version that > es-hadoop is compiled against matches the one used by the Spark version > compiled against for each release - typically this shouldn't pose a problem. > > Unfortunately, despite the minor version increments, some of the Spark > APIs or components (in particular Spark SQL) have changed drastically > between each release breaking backwards compatibility. For example, Beta3 > works until Spark 1.1 (which was the latest stable release during its > release) but not with 1.2. This is fixed in master however the current dev > build doesn't work with Spark SQL in the newly released 1.3 (does work with > Spark core). > > This has already been fixed locally however I'm having difficulties trying > to preserve compatibility across the Spark SQL 1.2 release and 1.3. > > Long story short, as long as the dependencies for Spark are in order, the > same should apply for es-hadoop as well since it relies only on Spark (and > Scala of course). > > On Tue, Mar 17, 2015 at 10:43 PM, Jeff Steinmetz \<[jeffrey....@gmail.com](mailto:jeffrey....@gmail.com) > \\> wrote: > > > There are plenty of spark / akka / scala / elasticsearch-hadoop > > dependencies to keep track of. > > > > Is it true that elasticsearch-hadoop needs to be compiled for a specific > > spark version to run correctly on the cluster? I'm also trying to keep > > track of the akka version and scala version. i.e, wil es-hadoop compiled > > for spark 1.2 work with Spark 1.3 ? > > > > When the elasticsearch-hadoop versions are released, as v2.0 v2.1, > > v2.1.0.Beta3, at what point do we need to keep in mind what spark version > > it was also compiled against? > > i.e., is it safe to assume the es-hadoop versions are tied to a specific > > spark core version? > > > > # I've been keeping the following chart in my notes to see what all the versions are with all dependencies > > > > Akka Version Dependencies > > Current Akka Stable Release: 2.3.9 > > > > Elasticsearch-Hadoop: 2.1.0Beta3 = Spark 1.1.0 > > Elasticsearch-Hadoop: 2.1.0Beta3-SNAPSHOT = Spark 1.2.1 > > Elasticsearch-Hadoop: what about spark 1.3 ? > > > > Spark: 1.3, Akka: 2.3.4-spark > > Spark: 1.2, Akka: 2.3.4-spark > > Spark: 1.1, Akka: 2.2.3-shaded-protobuf > > > > Activator 1.2.12 comes with with Akka 2.3.4 > > > > Play 2.3.8, akka 2.3.4, scala 2.11.1 (will also work with 2.10.4 ) > > Play 2.2.x, akka 2.2.0 > > > > Spark Job Server 0.4.1, Spark Core 1.1.0, Akka, 2.2.4 > > Spark Job Server Master as of Feb 22, 2015, Spark Core 1.2.0, Akka > > 2.3.4, Scala 2.10.4 > > > > Akka persistence latest 2.3.4 or later > > Akka 2.3.9 is released for Scala 2.10.4 and 2.11.5 > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/28ad3f78-8b3d-450a-a29d-06d3e6636cfd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/28ad3f78-8b3d-450a-a29d-06d3e6636cfd%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/28ad3f78-8b3d-450a-a29d-06d3e6636cfd%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/28ad3f78-8b3d-450a-a29d-06d3e6636cfd%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0c19d1fa-17b9-4e6e-a698-b49c7d6919d3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0c19d1fa-17b9-4e6e-a698-b49c7d6919d3%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [March 18, 2015, 12:56am UTC](https://discuss.elastic.co/t/spark-version-elasticsearch-hadoop-version-akka-version-sync-up/22722/4 "2015-03-18T00:56:01Z")
You're close: elasticsearch-hadoop snapshot (aka dev aka master) works on spark 1.2, 1.1 and 1.0, both core and sql elasticsearch-hadoop beta3 (not snapshot) works on spark 1.1 and spark 1.0, both core and sql elasticsearch-hadoop beta2 (not snapshot) works on spark 1.0 (core and sql) The support for spark 1.3 hasn't been committed yet, I'd like to push it out once I (hopefully) manage to keep compatibility on the sql integration with spark 1.2 and lower. Once it is, it will be available in the nightly/dev builds published (as the name implies) every 24th - each day or night, depending on your timezon [1]. > Costin - I am amazed by your ability to keep all this straight - my head > would explode dealing with all the dependencies in flux. Kudos to you. Thanks. As most (if not all) things in life, practice makes perfect 😉 [1] > **[Installation | Elasticsearch for Apache Hadoop \[master\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/hadoop/master/install.html#download-dev)** > > Reference documentation of elasticsearch-hadoop On Wed, Mar 18, 2015 at 12:59 AM, Jeff Steinmetz \< [jeffrey.steinmetz@gmail.com](mailto:jeffrey.steinmetz@gmail.com)\> wrote: > Thank you for the summary - you are confirming (as a sanity check for > myself): > > elasticsearch-hadoop beta3 (not snapshot) on spark core 1.1 only > elasticsearch-hadoop-beta3-SNAPSHOT with spark core 1.1, 1.2 and 1.3 -- as > long as I don't use Spark SQL when using 1.2 and 1.3 > > Costin - I am amazed by your ability to keep all this straight - my head > would explode dealing with all the dependencies in flux. Kudos to you. > > On Tuesday, March 17, 2015 at 2:12:06 PM UTC-7, Costin Leau wrote: > > > es-hadoop doesn't depend on akka, only on Spark. The scala version that > > es-hadoop is compiled against matches the one used by the Spark version > > compiled against for each release - typically this shouldn't pose a problem. > > > > Unfortunately, despite the minor version increments, some of the Spark > > APIs or components (in particular Spark SQL) have changed drastically > > between each release breaking backwards compatibility. For example, Beta3 > > works until Spark 1.1 (which was the latest stable release during its > > release) but not with 1.2. This is fixed in master however the current dev > > build doesn't work with Spark SQL in the newly released 1.3 (does work with > > Spark core). > > > > This has already been fixed locally however I'm having difficulties > > trying to preserve compatibility across the Spark SQL 1.2 release and 1.3. > > > > Long story short, as long as the dependencies for Spark are in order, the > > same should apply for es-hadoop as well since it relies only on Spark (and > > Scala of course). > > > > On Tue, Mar 17, 2015 at 10:43 PM, Jeff Steinmetz [jeffrey....@gmail.com](mailto:jeffrey....@gmail.com) > > wrote: > > > > > There are plenty of spark / akka / scala / elasticsearch-hadoop > > > dependencies to keep track of. > > > > > > Is it true that elasticsearch-hadoop needs to be compiled for a specific > > > spark version to run correctly on the cluster? I'm also trying to keep > > > track of the akka version and scala version. i.e, wil es-hadoop compiled > > > for spark 1.2 work with Spark 1.3 ? > > > > > > When the elasticsearch-hadoop versions are released, as v2.0 v2.1, > > > v2.1.0.Beta3, at what point do we need to keep in mind what spark version > > > it was also compiled against? > > > i.e., is it safe to assume the es-hadoop versions are tied to a specific > > > spark core version? > > > > > > # I've been keeping the following chart in my notes to see what all the versions are with all dependencies > > > > > > Akka Version Dependencies > > > Current Akka Stable Release: 2.3.9 > > > > > > Elasticsearch-Hadoop: 2.1.0Beta3 = Spark 1.1.0 > > > Elasticsearch-Hadoop: 2.1.0Beta3-SNAPSHOT = Spark 1.2.1 > > > Elasticsearch-Hadoop: what about spark 1.3 ? > > > > > > Spark: 1.3, Akka: 2.3.4-spark > > > Spark: 1.2, Akka: 2.3.4-spark > > > Spark: 1.1, Akka: 2.2.3-shaded-protobuf > > > > > > Activator 1.2.12 comes with with Akka 2.3.4 > > > > > > Play 2.3.8, akka 2.3.4, scala 2.11.1 (will also work with 2.10.4 ) > > > Play 2.2.x, akka 2.2.0 > > > > > > Spark Job Server 0.4.1, Spark Core 1.1.0, Akka, 2.2.4 > > > Spark Job Server Master as of Feb 22, 2015, Spark Core 1.2.0, Akka > > > 2.3.4, Scala 2.10.4 > > > > > > Akka persistence latest 2.3.4 or later > > > Akka 2.3.9 is released for Scala 2.10.4 and 2.11.5 > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/28ad3f78-8b3d-450a-a29d-06d3e6636cfd% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/28ad3f78-8b3d-450a-a29d-06d3e6636cfd%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/28ad3f78-8b3d-450a-a29d-06d3e6636cfd%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/0c19d1fa-17b9-4e6e-a698-b49c7d6919d3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0c19d1fa-17b9-4e6e-a698-b49c7d6919d3%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/0c19d1fa-17b9-4e6e-a698-b49c7d6919d3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0c19d1fa-17b9-4e6e-a698-b49c7d6919d3%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJogdmf9-Fs4xDH9wVoVxvOG3CMSZ\_XfCZb7%2BLZDT1AvDnxrZg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJogdmf9-Fs4xDH9wVoVxvOG3CMSZ_XfCZb7%2BLZDT1AvDnxrZg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:26am UTC](https://discuss.elastic.co/t/spark-version-elasticsearch-hadoop-version-akka-version-sync-up/22722/5 "2017-07-06T00:26:03Z")